Articles that discuss the use of LLMs in science.
| Title | Type of Resource | Description of Resource | Link to Resource | Open Science | Use of LLM | Research Discipline(s) |
|---|---|---|---|---|---|---|
| “Conversing” with Qualitative Data: Enhancing Qualitative Research through Large Language Models (LLMs) | Discussion Article | In this paper, I explore the transformative potential of Large Language Models (LLMs) such as ChatGPT in the realm of qualitative research, particularly in the social sciences. These generative AI models, trained on extensive textual data, have the unique ability to "understand," generate, and manipulate human-like text, offering unprecedented opportunities for data analysis and interpretation. I argue that LLMs, with this capacity, can significantly enhance the depth and efficiency of qualitative analysis. They can quickly identify patterns, themes, and sentiments in the data, providing a level of nuance that can be challenging to achieve with manual coding. Furthermore, their ability to generate human-like text can be used to simulate social interactions, create engaging presentations of research findings, and even "converse" with the data in a natural and flexible way. Indeed a central contribution of this paper lies in exploring this novel concept of "asking questions of" or "conversing with" text-based data, which opens up new avenues for qualitative research and analysis. This interactive capability of LLMs provides a transformative approach to topic coding and content analysis, allowing researchers to pose complex, nuanced questions to their data and receive responses in natural language. Ethical considerations and limitations are also discussed. | Conversing | Preprint | Data Analysis | Other |
| AI and the transformation of social science research | Discussion Article | Advances in artificial intelligence (AI), particularly large language models (LLMs), are substantially affecting social science research. These transformer-based machine-learning models pretrained on vast amounts of text data are increasingly capable of simulating human-like responses and behaviors (1, 2), offering opportunities to test theories and hypotheses about human behavior at great scale and speed. This presents urgent challenges: How can social science research practices be adapted, even reinvented, to harness the power of foundational AI? And how can this be done while ensuring transparent and replicable research? | Pay Walled Link (sorry, if I find a open access copy I will update) | Other | ||
| AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap | Discussion Article | The rise of powerful large language models (LLMs) brings about tremendous opportunities for innovation but also looming risks for individuals and society at large. We have reached a pivotal moment for ensuring that LLMs and LLM-infused applications are developed and deployed responsibly. However, a central pillar of responsible AI—transparency—is largely missing from the current discourse around LLMs. It is paramount to pursue new approaches to provide transparency for LLMs, and years of research at the intersection of AI and human-computer interaction (HCI) highlight that we must do so with a human-centered perspective: Transparency is fundamentally about supporting appropriate human understanding, and this understanding is sought by different stakeholders with different goals in different contexts. In this new era of LLMs, we must develop and design approaches to transparency by considering the needs of stakeholders in the emerging LLM ecosystem, the novel types of LLM-infused applications being built, and the new usage patterns and challenges around LLMs, all while building on lessons learned about how people process, interact with, and make use of information. We reflect on the unique challenges that arise in providing transparency for LLMs, along with lessons learned from HCI and responsible AI research that has taken a human-centered perspective on AI transparency. We then lay out four common approaches that the community has taken to achieve transparency—model reporting, publishing evaluation results, providing explanations, and communicating uncertainty—and call out open questions around how these approaches may or may not be applied to LLMs. We hope this provides a starting point for discussion and a useful roadmap for future research. | LLM Transparency | Preprint | Other | |
| Science in the Era of ChatGPT, Large Language Models and AI: Challenges for Research Ethics Review and How to Respond | Discussion Article | Large language models of artificial intelligence (AI) such as ChatGPT find remarkable but controversial applicability in science and research. This paper reviews epistemological challenges, ethical and integrity risks in science conduct. This is with the aim to lay new timely foundations for a high-quality research ethics review in the era of AI. The role of AI language models as a research instrument and subject is scrutinized along with ethical implications for scientists, participants and reviewers. Ten recommendations shape a response for a more responsible research conduct with AI language models. | Ethics of using AI in research | Preprint | Other | |
| Towards Human-AI Collaborative Urban Science Research Enabled by Pre-trained Large Language Models | Discussion Article | Pre-trained large language models (PLMs) have the potential to support urban science research through content creation, information extraction, assisted programming, text classification, and other technical advances. In this research, we explored the opportunities, challenges, and prospects of PLMs in urban science research. Specifically, we discussed potential applications of PLMs to urban institution, urban space, urban information, and citizen behaviors research through seven examples using ChatGPT. We also examined the challenges of PLMs in urban science research from both technical and social perspectives. The prospects of the application of PLMs in urban science research were then proposed. We found that PLMs can effectively aid in understanding complex concepts in urban science, facilitate urban spatial form identification, assist in disaster monitoring, and sense public sentiment. At the same time, however, the applications of PLMs in urban science research face evident threats, such as technical limitations, security, privacy, and social bias. The development of fundamental models based on domain knowledge and human-AI collaboration may help improve PLMs to support urban science research in future. | LLM in urban science research | Preprint | Other | Geography, Sociology, Urban Planning, Other |
| Guidance for researchers and peer‑reviewers on the ethical use of Large Language Models (LLMs) in scientific research workflows | Discussion Article | For researchers interested in exploring the exciting applications of Large Language Models (LLMs) in their scientific investigations, there is currently limited guidance and few norms for them to consult. Similarly, those providing peer-reviews on research articles where LLMs were used are without conventions or standards to apply or guidelines to follow. This situation is understandable given the rapid and recent development of LLMs that are capable of valuable contributions to research workflows (such as OpenAI’s ChatGPT). Nevertheless, now is the time to begin the development of norms, conventions, and standards that can be applied by researchers and peer-reviewers. By applying the principles of Artificial Intelligence (AI) ethics, we can better ensure that the use of LLMs in scientific research aligns with ethical principles and best practices. This editorial hopes to inspire further dialogue and research in this crucial area of scientific investigation | Norms and Review Criteria | |||
| Can Large Language Models Transform Computational Social Science? | Discussion Article | Large Language Models (LLMs) like ChatGPT are capable of successfully performing many language processing tasks zero-shot (without the need for training data). If this capacity also applies to the coding of social phenomena like persuasiveness and political ideology, then LLMs could effectively transform Computational Social Science (CSS). This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 24 representative CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but still achieve fair levels of agreement with humans. On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers' gold references. We conclude that today's LLMs can radically augment the CSS research pipeline in two ways: (1) serving as zero-shot data annotators on human annotation teams, and (2) bootstrapping challenging creative generation tasks (e.g., explaining the hidden meaning behind text). In summary, LLMs can significantly reduce costs and increase efficiency of social science analysis in partnership with humans. | Coding Data | Preprint | Computer Science | |
| Can Generative AI Improve Social Science? | Discussion Article | Artificial intelligence that can produce realistic text, images, and other human-like outputs is currently transforming many different industries. Yet it is not yet known how such tools might transform social science research. In the first section of this article, I assess the potential of Generative AI to improve online experiments, agent-based models, and automated content analyses. I also discuss whether these tools may help social scientists perform literature reviews, identify novel research questions, and develop hypotheses to explain them. Next, I evaluate whether Generative AI can help social scientists with more mundane tasks such as acquiring advanced programming skills or writing more effective prose. In the second section of this article I discuss the limitations of Generative AI as well as how these tools might be employed by researchers in an ethical manner. I discuss how bias in the processes and data used to train these tools can negatively impact social science research as well as a range of other challenges related to accuracy, reproducibility, interpretability, and efficiency. I conclude by highlighting the need for increased collaboration between social scientists and artificial intelligence researchers--- not only to ensure that such tools are used in a safe and ethical manner, but also because the progress of artificial intelligence may require deeper understanding of theories of human behavior | Improve Social Sciences | Preprint | Other | |
| Language Models and Cognitive Automation for Economic Research | Discussion Article | Large language models (LLMs) such as ChatGPT have the potential to revolutionize research in economics and other disciplines. I describe 25 use cases along six domains in which LLMs are starting to become useful as both research assistants and tutors: ideation, writing, background research, data analysis, coding, and mathematical derivations. I provide general instructions and demonstrate specific examples for how to take advantage of each of these, classifying the LLM capabilities from experimental to highly useful. I hypothesize that ongoing advances will improve the performance of LLMs across all of these domains, and that economic researchers who take advantage of LLMs to automate micro tasks will become significantly more productive. Finally, I speculate on the longer-term implications of cognitive automation via LLMs for economic research. | www.nber.org/papers/w30957 | Open Source | Economics |