Articles that discuss the use of LLMs in science.
| Title | Type of Resource | Link to Resource | Date Recorded | Open Science | Use of LLM | Research Discipline(s) | Description of Resource |
|---|---|---|---|---|---|---|---|
| Can Large Language Models Transform Computational Social Science? | Discussion Article | Coding Data | May 13, 2023 | Preprint | Computer Science | Large Language Models (LLMs) like ChatGPT are capable of successfully performing many language processing tasks zero-shot (without the need for training data). If this capacity also applies to the coding of social phenomena like persuasiveness and political ideology, then LLMs could effectively transform Computational Social Science (CSS). This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 24 representative CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but still achieve fair levels of agreement with humans. On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers' gold references. We conclude that today's LLMs can radically augment the CSS research pipeline in two ways: (1) serving as zero-shot data annotators on human annotation teams, and (2) bootstrapping challenging creative generation tasks (e.g., explaining the hidden meaning behind text). In summary, LLMs can significantly reduce costs and increase efficiency of social science analysis in partnership with humans. | |
| Can Generative AI Improve Social Science? | Discussion Article | Improve Social Sciences | May 13, 2023 | Preprint | Other | Artificial intelligence that can produce realistic text, images, and other human-like outputs is currently transforming many different industries. Yet it is not yet known how such tools might transform social science research. In the first section of this article, I assess the potential of Generative AI to improve online experiments, agent-based models, and automated content analyses. I also discuss whether these tools may help social scientists perform literature reviews, identify novel research questions, and develop hypotheses to explain them. Next, I evaluate whether Generative AI can help social scientists with more mundane tasks such as acquiring advanced programming skills or writing more effective prose. In the second section of this article I discuss the limitations of Generative AI as well as how these tools might be employed by researchers in an ethical manner. I discuss how bias in the processes and data used to train these tools can negatively impact social science research as well as a range of other challenges related to accuracy, reproducibility, interpretability, and efficiency. I conclude by highlighting the need for increased collaboration between social scientists and artificial intelligence researchers--- not only to ensure that such tools are used in a safe and ethical manner, but also because the progress of artificial intelligence may require deeper understanding of theories of human behavior | |
| Language Models and Cognitive Automation for Economic Research | Discussion Article | www.nber.org/papers/w30957 | April 11, 2023 | Open Source | Economics | Large language models (LLMs) such as ChatGPT have the potential to revolutionize research in economics and other disciplines. I describe 25 use cases along six domains in which LLMs are starting to become useful as both research assistants and tutors: ideation, writing, background research, data analysis, coding, and mathematical derivations. I provide general instructions and demonstrate specific examples for how to take advantage of each of these, classifying the LLM capabilities from experimental to highly useful. I hypothesize that ongoing advances will improve the performance of LLMs across all of these domains, and that economic researchers who take advantage of LLMs to automate micro tasks will become significantly more productive. Finally, I speculate on the longer-term implications of cognitive automation via LLMs for economic research. |