Resource Hub

Explore resources on using LLMs in scientific research. Choose a topic, search by keyword, or combine filters to narrow the list.

Date recorded is the date the original site recorded an entry, not its publication date. Missing dates are shown as “Not recorded.” This catalog contains the listings recovered from the archived website; it is not the complete original database.

Download the resource CSV

34 resources

ResourceTypeDiscipline and useDate recorded
14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon
Description and source

Large-language models (LLMs) such as GPT-4 caught the interest of many scientists. Recent studies suggested that these models could be useful in chemistry and materials science. To explore these possibilities, we organized a hackathon. This article chronicles the projects built as part of this hackathon. Participants employed LLMs for various applications, including predicting properties of molecules and materials, designing novel interfaces for tools, extracting knowledge from unstructured data, and developing new educational applications. The diverse topics and the fact that working prototypes could be generated in less than two days highlight that LLMs will profoundly impact the future of our fields. The rich collection of ideas and projects also indicates that the applications of LLMs are not limited to materials science and chemistry but offer potential benefits to a wide range of scientific disciplines.

Open science: Preprint

Archived source 1

 Use Case Example Chemistry; Engineering
LLM use: Other
A Primer for Evaluating Large Language Models in Social Science Research
Description and source

Autoregressive Large Language Models (LLMs) exhibit remarkable conversational and reasoning abilities, and exceptional flexibility across a wide range of tasks. Subsequently, LLMs are being increasingly used in scientific research, to analyze data, generate synthetic data, or even to write scientific papers. This trend necessitates that authors follow best practices for conducting and reporting LLM research and that journal reviewers are able to evaluate the quality of works that use LLMs. We provide authors of social scientific research with essential recommendations to ensure replicable and robust results using LLMs. Our recommendations also highlight considerations for reviewers, focusing on methodological rigor, replicability, and validity of results when evaluating studies that use LLMs to automate data processing or simulate human data. We offer practical advice on assessing the appropriateness of LLM applications in submitted studies, emphasizing the need for transparency in methodological reporting and the challenges posed by the non-deterministic and continuously evolving nature of these models. By providing a framework for best practices and critical review, this primer aims to ensure high-quality, innovative research within the evolving landscape of social science studies using LLMs.

Open science: Preprint

Archived source 1

 Discussion Article Psychology
LLM use: Other
AI-Augmented Cultural Sociology: Guidelines for LLM-assisted text analysis and an illustrative example
Description and source

The advent of large language models (LLMs) presents a promising opportunity for how we analyze text and, by extension, can study the role of culture and symbolic meanings in social life. Using an illustrative example focused on the concept of “personalized service” within Michelin-starred restaurants, this research note demonstrates how LLMs can reliably identify complex, multifaceted concepts similarly to a qualitative data analyst, but in a more scalable manner. We extend existing validation approaches, offering guidelines on the amount of manually coded data needed to evaluate LLM-generated outputs, drawing on sampling theory and a data simulation. We also discuss broader applications of LLMs in cultural sociology, such as investigations on established concepts (e.g., cultural consecration) and emerging concepts (e.g., future-oriented deliberation). This discussion underscores that AI-tools can significantly augment the empirical scope of research projects, building on rather than replacing traditional qualitative approaches. Our study ultimately advocates for an optimistic yet cautious engagement with AI-tools in social scientific inquiry, highlighting both their analytic potential and the need for ongoing reflection on their ethical implications.

Open science: Preprint

Archived source 1

 Research Article; Use Case Example Sociology
LLM use: Data Analysis
AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction
Description and source

Large language models (LLMs) that produce human-like responses have begun to revolutionize research practices in the social sciences. This paper shows how we can integrate LLMs and social surveys to accurately predict individual responses to survey questions that were not asked before. We develop a novel methodological framework to personalize LLMs by considering the meaning of survey questions derived from their text, the latent beliefs of individuals inferred from their response patterns, and the temporal contexts across different survey periods through fine-tuning LLMs with survey data. Using the General Social Survey from 1972 to 2021, we show that the fine-tuned model based on Alpaca-7b can predict individual responses to survey questions that are partially missing as well as entirely missing. The remarkable prediction capabilities allow us to fill in missing trends with high confidence and pinpoint when public attitudes changed, such as the rising support for same-sex marriage. We discuss practical constraints, socio-demographic representation, and ethical concerns regarding individual autonomy and privacy when using LLMs for opinion prediction. This study demonstrates that LLMs and surveys can mutually enhance each other's capabilities: LLMs broaden survey potential, while surveys improve the alignment of LLMs.

Open science: Preprint

Archived source 1

 Research Article Computer Science
LLM use: Data Collection; Data Cleaning/Preparation		 Not recorded
Artificial intelligence and illusions of understanding in scientific research
Description and source

Scientists are enthusiastically imagining ways in which artificial intelligence (AI) tools might improve research. Why are AI tools so attractive and what are the risks of implementing them across the research pipeline? Here we develop a taxonomy of scientists’ visions for AI, observing that their appeal comes from promises to improve productivity and objectivity by overcoming human shortcomings. But proposed AI solutions can also exploit our cognitive limitations, making us vulnerable to illusions of understanding in which we believe we understand more about the world than we actually do. Such illusions obscure the scientific community’s ability to see the formation of scientific monocultures, in which some types of methods, questions and viewpoints come to dominate alternative approaches, making science less innovative and more vulnerable to errors. The proliferation of AI tools in science risks introducing a phase of scientific enquiry in which we produce more but understand less. By analysing the appeal of these tools, we provide a framework for advancing discussions of responsible knowledge production in the age of AI.

Open science: Open Source

Archived source 1

 Discussion Article Any Discipline
LLM use: Research Design; Other
Autonomous chemical research with large language models
Description and source

Transformer-based large language models are making significant strides in various fields, such as natural language processing1,2,3,4,5, biology6,7, chemistry8,9,10 and computer programming11,12. Here, we show the development and capabilities of Coscientist, an artificial intelligence system driven by GPT-4 that autonomously designs, plans and performs complex experiments by incorporating large language models empowered by tools such as internet and documentation search, code execution and experimental automation. Coscientist showcases its potential for accelerating research across six diverse tasks, including the successful reaction optimization of palladium-catalysed cross-couplings, while exhibiting advanced capabilities for (semi-)autonomous experimental design and execution. Our findings demonstrate the versatility, efficacy and explainability of artificial intelligence systems like Coscientist in advancing research.

Open science: Open Source

Archived source 1

 Discussion Article Chemistry
LLM use: Other
Challenges in Guardrailing Large Language Models for Science
Description and source

The rapid development in large language models (LLMs) has transformed the landscape of natural language processing and understanding (NLP/NLU), offering significant benefits across various domains. However, when applied to scientific research, these powerful models exhibit critical failure modes related to scientific integrity and trustworthiness. Existing general-purpose LLM guardrails are insufficient to address these unique challenges in the scientific domain. We propose a comprehensive taxonomic framework for LLM guardrails encompassing four key dimensions: trustworthiness, ethics & bias, safety, and legal compliance. Our framework includes structured implementation guidelines for scientific research applications, incorporating white-box, blackbox, and gray-box methodologies. This approach specifically addresses critical challenges in scientific LLM deployment, including temporal sensitivity, knowledge contextualization, conflict resolution, and intellectual property protection.

Open science: Preprint

Archived source 1

 Discussion Article Any Discipline
LLM use: Research Design; Other
Contribution and Challenges of ChatGPT and Similar Generative Artificial Intelligence in Biochemistry, Genetics and Molecular Biology
Description and source

The incorporation of ChatGPT, an advanced natural language processing model, into the realms of biochemistry, genetics, and molecular biology has revolutionized research and communication within these fields. This study explores the impacts and obstacles associated with ChatGPT in these domains. ChatGPT has made substantial contributions to the accessibility and dissemination of knowledge in biochemistry, genetics, and molecular biology. It simplifies complex scientific literature, offers concise explanations, answers queries, and generates summaries, benefiting researchers, students, and practitioners. Furthermore, it fosters global collaboration by enabling discussions and knowledge sharing among scientists. One of the primary advantages of ChatGPT is its assistance in decoding intricate genomic and proteomic data. It aids in genetic sequence analysis, identifies potential disease markers, and provides suggestions for experimental designs. Additionally, ChatGPT can assist in composing and reviewing research papers, elevating the quality of scientific publications in these fields. However, despite its merits, ChatGPT encounters challenges in the context of biochemistry, genetics, and molecular biology. It may struggle to grasp highly specialized or novel research topics, potentially leading to the dissemination of inaccurate information if not used judiciously. Privacy concerns arise when discussing sensitive genetic or medical data. ChatGPT brings valuable advantages to the domains of biochemistry, genetics, and molecular biology by simplifying information access, promoting collaborative research, and aiding in data interpretation. Nevertheless, users must remain vigilant about potential inaccuracies and privacy issues. Addressing these challenges as technology evolves will be crucial to fully unlock ChatGPT's potential in advancing research and education within these critical scientific disciplines.

Open science: Preprint

Archived source 1

 Discussion Article Biology
LLM use: Other
Emergent autonomous scientific research capabilities of large language models
Description and source

Transformer-based large language models are rapidly advancing in the field of machine learning research, with applications spanning natural language, biology, chemistry, and computer programming. Extreme scaling and reinforcement learning from human feedback have significantly improved the quality of generated text, enabling these models to perform various tasks and reason about their choices. In this paper, we present an Intelligent Agent system that combines multiple large language models for autonomous design, planning, and execution of scientific experiments. We showcase the Agent's scientific research capabilities with three distinct examples, with the most complex being the successful performance of catalyzed cross-coupling reactions. Finally, we discuss the safety implications of such systems and propose measures to prevent their misuse.

Open science: Preprint

Archived source 1

 Research Article Computer Science
LLM use: Other		 Not recorded
Engineering of Inquiry: The “Transformation” of Social Science through Generative AI
Description and source

We increasingly read that generative AI will “transform” the social sciences, but little to no work has conceptualized the conditions necessary to fulfill such a promise. We review recent research on generative AI and evaluate its potential to reshape research practices. As the technology advances, generative AI could support various research tasks, including idea generation, data collection, and analysis. However, we discuss three challenges to an optimistic outlook that focuses solely on accelerating research through practical tools and reducing costs through inexpensive “synthetic” data. First, generative AI raises severe concerns about the validity of conclusions drawn from synthetic data about human populations. Second, possible efficiency gains in the research process may be partially offset by new problems introduced by the technology. Third, applications of generative AI have so far focused on enhancing existing methods, with limited efforts to harness the technology’s unique potential to simulate human behavior in social environments. Sociologists could use sociological theories and methods to develop “generative agents.” A new “trading zone” could emerge where social scientists, statisticians, and computer scientists develop new methodologies to facilitate innovative lines of inquiry and produce scientifically valid conclusions.

Open science: Preprint

Archived source 1

 Discussion Article Any Discipline
LLM use: Research Design; Other
Generative AI Can Supercharge Your Academic Research
Description and source

Conducting relevant scholarly research can be a struggle. Educators must employ innovative research methods, carefully analyze complex data, and then master the art of writing clearly, all while keeping the interest of a broad audience in mind. Generative AI is revolutionizing this sometimes tedious aspect of academia by providing sophisticated tools to help educators navigate and elevate their research. But there are concerns, too. AI’s capabilities are rapidly expanding into areas that were once considered exclusive to humans, like creativity and ingenuity. This could lead to improved productivity, but it also raises questions about originality, data manipulation, and credibility in research. With a simple prompt, AI can easily generate falsified datasets, mimic others’ research, and avoid plagiarism detection. [4 how to tutorials follow]

Open science: Open Source

Archived source 1

 Discussion Article; Use Case Example Other
LLM use: Other
Generative AI for Economic Research: LLMs Learn to Collaborate and Reason
Description and source

Large language models (LLMs) have seen remarkable progress in speed, cost efficiency, accuracy, and the capacity to process larger amounts of text over the past year. This article is a practical guide to update economists on how to use these advancements in their research. The main innovations covered are (i) new reasoning capabilities, (ii) novel workspaces for interactive LLM collaboration such as Claude's Artifacts, ChatGPT's Canvas or Microsoft's Copilot, and (iii) recent improvements in LLM-powered internet search. Incorporating these capabilities in their work allows economists to achieve significant productivity gains. Additionally, I highlight new use cases in promoting research, such as automatically generated blog posts, presentation slides and interviews as well as podcasts via Google's NotebookLM.

Open science: Open Source

Archived source 1

 Discussion Article; Use Case Example Economics
LLM use: Other
Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations
Description and source

Generative artificial intelligence (GenAI) offers promising potential for advancing human-AI collaboration in qualitative research. However, existing works focused on conventional machine-learning and pattern-based AI systems, and little is known about how researchers interact with GenAI in qualitative research. This work delves into researchers' perceptions of their collaboration with GenAI, specifically ChatGPT. Through a user study involving ten qualitative researchers, we found ChatGPT to be a valuable collaborator for thematic analysis, enhancing coding efficiency, aiding initial data exploration, offering granular quantitative insights, and assisting comprehension for non-native speakers and non-experts. Yet, concerns about its trustworthiness and accuracy, reliability and consistency, limited contextual understanding, and broader acceptance within the research community persist. We contribute five actionable design recommendations to foster effective human-AI collaboration. These include incorporating transparent explanatory mechanisms, enhancing interface and integration capabilities, prioritising contextual understanding and customisation, embedding human-AI feedback loops and iterative functionality, and strengthening trust through validation mechanisms.

Open science: Preprint

Archived source 1

 Research Article Computer Science
LLM use: Data Analysis		 Not recorded
LangChain Cookbook
Description and source

Goal: Provide an introductory understanding of the components and use cases of LangChain via ELI5 examples and code snippets.

Open science: Open Source

Archived source 1

 Tutorial w/ Code Not recorded
LLM use: Not recorded		 Not recorded
LangChain Library Documentation
Description and source

LangChain is a framework for developing applications powered by language models. We believe that the most powerful and differentiated applications will not only call out to a language model via an API, but will also: Be data-aware: connect a language model to other sources of data Be agentic: allow a language model to interact with its environment The LangChain framework is designed with the above principles in mind.

Open science: Open Source

Archived source 1

 Documentation Not recorded
LLM use: Not recorded		 Not recorded
Leveraging generative artificial intelligence to simulate student learning behavior
Description and source

Student simulation presents a transformative approach to enhance learning outcomes, advance educational research, and ultimately shape the future of effective pedagogy. We explore the feasibility of using large language models (LLMs), a remarkable achievement in AI, to simulate student learning behaviors. Unlike conventional machine learning based prediction, we leverage LLMs to instantiate virtual students with specific demographics and uncover intricate correlations among learning experiences, course materials, understanding levels, and engagement. Our objective is not merely to predict learning outcomes but to replicate learning behaviors and patterns of real students. We validate this hypothesis through three experiments. The first experiment, based on a dataset of N = 145, simulates student learning outcomes from demographic data, revealing parallels with actual students concerning various demographic factors. The second experiment (N = 4524) results in increasingly realistic simulated behaviors with more assessment history for virtual students modelling. The third experiment (N = 27), incorporating prior knowledge and course interactions, indicates a strong link between virtual students' learning behaviors and fine-grained mappings from test questions, course materials, engagement and understanding levels. Collectively, these findings deepen our understanding of LLMs and demonstrate its viability for student simulation, empowering more adaptable curricula design to enhance inclusivity and educational effectiveness.

Open science: Preprint

Archived source 1

 Research Article Data Science; Education
LLM use: Data Generation; Data Analysis		 Not recorded
LLMs and the Risk of Sloppy Science: Navigating the Future of Scientific Inquiry in the Age of Artificial Intelligence
Description and source

The emergence of Large Language Models (LLMs) such as GPT-3 represents a paradigm shift in scientific research, offering unparalleled capabilities in data analysis, hypothesis generation, and literature synthesis. However, their integration into research processes raises fundamental philosophical, methodological and ethical concerns. This article critically examines the benefits and risks associated with LLMs through the lenses of philosophy of science, cognitive science, and second-order cybernetics in order to augment a public debate that has until now seemed excessively focused on practical implementation risks rather than categorical errors. We explore the tension between augmenting human research capabilities and the threat of "sloppy science," where the ease of generating scientific content may compromise the quality and reliability of research outputs.

Open science: Preprint

Archived source 1

 Discussion Article Not recorded
LLM use: Other
LLMs for Science: Usage for Code Generation and Data Analysis
Description and source

Large language models (LLMs) have been touted to enable increased productivity in many areas of today's work life. Scientific research as an area of work is no exception: the potential of LLM-based tools to assist in the daily work of scientists has become a highly discussed topic across disciplines. However, we are only at the very onset of this subject of study. It is still unclear how the potential of LLMs will materialise in research practice. With this study, we give first empirical evidence on the use of LLMs in the research process. We have investigated a set of use cases for LLM-based tools in scientific research, and conducted a first study to assess to which degree current tools are helpful. In this paper we report specifically on use cases related to software engineering, such as generating application code and developing scripts for data analytics. While we studied seemingly simple use cases, results across tools differ significantly. Our results highlight the promise of LLM-based tools in general, yet we also observe various issues, particularly regarding the integrity of the output these tools provide.

Open science: Preprint; Open Source; Open Data; Open Code

Archived source 1

 Research Article Computer Science
LLM use: Data Generation; Data Analysis		 Not recorded
Machine Learning as a Tool for Hypothesis Generation
Description and source

While hypothesis testing is a highly formalized activity, hypothesis generation remains largely informal. We propose a procedure that uses machine learning algorithms—and their capacity to notice patterns people might not—to generate novel hypotheses about human behavior. We illustrate the procedure with a concrete application: judge decisions. We begin with a striking fact: up to half of the predictable variation in who judges jail is explained solely by the pixels in the defendant’s mugshot—that is, the predictions from an algorithm built using just facial images. We develop a procedure that allows human subjects to interact with this black-box algorithm to produce hypotheses about what in the face influences judge decisions. The procedure generates hypotheses that are both interpretable and novel: They are not explained by factors implied by existing research (demographics, facial features emphasized by previous psychology studies), nor are they already known (even if just tacitly) to people or even experts. Though these results are specific, our procedure is general. It provides a way to produce novel, interpretable hypotheses from any high-dimensional dataset (e.g. cell phones, satellites, online behavior, news headlines, corporate filings, and high-frequency time series). A central tenet of our paper is that hypothesis generation is in and of itself a valuable activity, and hope this encourages future work in this largely “pre-scientific” stage of science.

Open science: Preprint

Archived source 1

 Research Article Economics
LLM use: Other		 Not recorded
Mathematical discoveries from program search with large language models
Description and source

Large Language Models (LLMs) have demonstrated tremendous capabilities in solving complex tasks, from quantitative reasoning to understanding natural language. However, LLMs sometimes suffer from confabulations (or hallucinations) which can result in them making plausible but incorrect statements [1,2]. This hinders the use of current large models in scientific discovery. Here we introduce FunSearch (short for searching in the function space), an evolutionary procedure based on pairing a pre-trained LLM with a systematic evaluator. We demonstrate the effectiveness of this approach to surpass the best known results in important problems, pushing the boundary of existing LLM-based approaches [3]. Applying FunSearch to a central problem in extremal combinatorics — the cap set problem — we discover new constructions of large cap sets going beyond the best known ones, both in finite dimensional and asymptotic cases. This represents the first discoveries made for established open problems using LLMs. We showcase the generality of FunSearch by applying it to an algorithmic problem, online bin packing, finding new heuristics that improve upon widely used baselines. In contrast to most computer search approaches, FunSearch searches for programs that describe how to solve a problem, rather than what the solution is. Beyond being an effective and scalable strategy, discovered programs tend to be more interpretable than raw solutions, enabling feedback loops between domain experts and FunSearch, and the deployment of such programs in real-world applications.

Open science: Open Source

Archived source 1

 Research Article Math
LLM use: Data Generation; Data Analysis; Other		 Not recorded
Named Entity Recognition for peer-review disambiguation in academic publishing
Description and source

In recent years, there has been a constant increase in the number of scientific peer-reviewed articles published. Each of these articles has to go through a laborious process, from peer review, through author revision rounds, to the final decision made by the editor-in-chief. Lacking time and being under pressure with diverse research tasks, senior scientists need new tools to automate parts of their activities. In this paper, we propose a new approach based on named entity recognition that is able to annotate review comments in order to extract meaningful information about changes requested by reviewers. This research focuses on deep learning models that are achieving state-of-the-art results in many natural language processing tasks. Exploring the performance of BERT-based and XLNet models on the review comments annotation task, a “review-annotation“ model based on SciBERT was trained, able to achieve an F1 score of 0.87. Its usage allows different players in the academic publishing process to better understand the review request. In addition, the correlation of the requested and the actual changes is made possible, allowing the final decision-maker to strengthen the article evaluation.

Open science: Not recorded

Archived source 1

 Research Article Computer Science; Education
LLM use: Not recorded		 Not recorded
OpenAI API Documentation
Description and source

The OpenAI API can be applied to virtually any task that involves understanding or generating natural language, code, or images. We offer a spectrum of models with different levels of power suitable for different tasks, as well as the ability to fine-tune your own custom models. These models can be used for everything from content generation to semantic search and classification.

Open science: Open Source

Archived source 1

 Documentation Not recorded
LLM use: Not recorded		 Not recorded
OpenAI Cookbook
Description and source

The OpenAI Cookbook shares example code for accomplishing common tasks with the OpenAI API. To run these examples, you'll need an OpenAI account and associated API key (create a free account). Most code examples are written in Python, though the concepts can be applied in any language.

Open science: Open Source; Open Code

Archived source 1 Archived source 2

 Documentation; Tutorial w/ Code Not recorded
LLM use: Not recorded		 Not recorded
Quality of Large Language Model Responses to Radiation Oncology Patient Care Questions
Description and source

This publications outlines a comprehensive evaluation approach to determine an LLM’s quality of responses to radiation oncology patient care questions using both domain-specific expertise and domain-agnostic metrics. Domain-specific expertise involves evaluating LLM-generated responses through human expert assessments, augmented with Likert scales, while domain-agnostic evaluation utilizes computational quantitative techniques to assess the LLM-generated responses.

Open science: Not recorded

Archived source 1

 Research Article; Application/Tool; Use Case Example Medicine; Public Health
LLM use: Research Design; Science Communication; Other
Questions of science: chatting with ChatGPT about complex systems
Description and source

We present an overview of the complex systems field using ChatGPT as a representation of the community's understanding. ChatGPT has learned language patterns and styles from a large dataset of internet texts, allowing it to provide answers that reflect common opinions, ideas, and language patterns found in the community. Our exploration covers both teaching and learning, and research topics. We recognize the value of ChatGPT as a source for the community's ideas.

Open science: Preprint

Archived source 1

 Research Article; Use Case Example Other
LLM use: Data Collection
Reaching the Gold Standard: Automated Text Analysis with Generative Pre-trained Transformers Matches Human-Level Performance
Description and source

Natural language is a vital source of evidence for the social sciences. Yet quantifying large volumes of text rigorously and precisely is extremely difficult, and automated methods have struggled to match the “gold standard” of human coding. The present work used GPT-4 to conduct an automated analysis of 1,356 essays, rating the authors’ spirituality on a continuous scale. This presents an especially challenging test for automated methods, due to the subtlety of the concept and the difficulty of inferring complex personality traits from a person’s writing. Nonetheless, we found that GPT-4’s ratings demonstrated excellent internal reliability, remarkable consistency with a human rater, and strong correlations with self-report measures and behavioral indicators of spirituality. These results suggest that, even on nuanced tasks requiring a high degree of conceptual sophistication, automated text analysis with Generative Pre-trained Transformers can match human-level performance. Hence, these results demonstrate the extraordinary potential for such tools to advance social scientific research.

Open science: Preprint

Archived source 1

 Research Article Computer Science
LLM use: Data Analysis		 Not recorded
Techniques for supercharging academic writing with generative AI
Description and source

Generalist large language models can elevate the quality and efficiency of academic writing.

Open science: Preprint

Archived source 1

 Documentation; Tutorial w/ Code; Tutorial w/o Code; Application/Tool; Discussion Article; Use Case Example; Reporting Guidelines Any Discipline
LLM use: Describing Results; Science Communication
The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery [Github Repo]
Description and source

One of the grand challenges of artificial intelligence is developing agents capable of conducting scientific research and discovering new knowledge. While frontier models have already been used to aid human scientists, e.g. for brainstorming ideas or writing code, they still require extensive manual supervision or are heavily constrained to a specific task. We're excited to introduce The AI Scientist, the first comprehensive system for fully automatic scientific discovery, enabling Foundation Models such as Large Language Models (LLMs) to perform research independently.

Open science: Open Source

Archived source 1

 Research Article; Tutorial w/ Code; Application/Tool; Use Case Example Computer Science
LLM use: Data Generation; Data Analysis; Science Communication
The Value of Generative AI for Qualitative Research: A Pilot Study
Description and source

This mixed-methods approach study investigates the potential of introducing generative AI (ChatGPT 4 and Bard) as part of a deductive qualitative research design that requires coding, focusing on possible gains in cost-effectiveness, coding throughput time, and inter-coder reliability (Cohen’s Kappa). This study involved semi-structured interviews with five domain experts and analyzed a dataset of 122 respondents that required categorization into six predefined categories. The results from using generative AI coders were compared with those from a previous study where human coders carried out the same task. In this comparison, we evaluated the performance of AI-based coders against two groups of human coders, comprising three experts and three non-experts. Our findings support the replacement of human coders with generative AI ones, specifically ChatGPT for deductive qualitative research methods of limited scope. The experimental group, consisting of three independent generative AI coders, outperformed both control groups in coding effort, with a fourfold (4x) efficiency and throughput time (15x) advantage. The latter could be explained by leveraging parallel processing. Concerning expert vs. non-expert coders, minimal evidence suggests a preference for experts. Although experts code slightly faster (17%), their inter-coder reliability showed no substantial advantage. A hybrid approach, combining ChatGPT and domain experts, shows the most promise. This approach reduces costs, shortens project timelines, and enhances inter-coder reliability, as indicated by higher Cohen’s Kappa values. In conclusion, generative AI, exemplified by ChatGPT, offers a viable alternative to human coders, in combination with human research involvement, delivering cost savings and faster research completion without sacrificing notable reliability. These insights, while limited in scope, show potential for further studies with larger datasets, more inductive qualitative research designs, and other research domains.

Open science: Open Source

Archived source 1

 Research Article; Use Case Example Data Science
LLM use: Data Analysis
Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation
Description and source

With the advent of large multimodal language models, science is now at a threshold of an AI-based technological transformation. Recently, a plethora of new AI models and tools has been proposed, promising to empower researchers and academics worldwide to conduct their research more effectively and efficiently. This includes all aspects of the research cycle, especially (1) searching for relevant literature; (2) generating research ideas and conducting experimentation; generating (3) text-based and (4) multimodal content (e.g., scientific figures and diagrams); and (5) AI-based automatic peer review. In this survey, we provide an in-depth overview over these exciting recent developments, which promise to fundamentally alter the scientific research process for good. Our survey covers the five aspects outlined above, indicating relevant datasets, methods and results (including evaluation) as well as limitations and scope for future research. Ethical concerns regarding shortcomings of these tools and potential for misuse (fake science, plagiarism, harms to research integrity) take a particularly prominent place in our discussion. We hope that our survey will not only become a reference guide for newcomers to the field but also a catalyst for new AI-based initiatives in the area of "AI4Science".

Open science: Preprint

Archived source 1

 Research Article; Discussion Article Computer Science; Any Discipline
LLM use: Research Design; Science Communication; Other
Tweetorial Hooks: Generative AI Tools to Motivate Science on Social Media (science communication)
Description and source

Communicating science and technology is essential for the public to understand and engage in a rapidly changing world. Tweetorials are an emerging phenomenon where experts explain STEM topics on social media in creative and engaging ways. However, STEM experts struggle to write an engaging "hook" in the first tweet that captures the reader's attention. We propose methods to use large language models (LLMs) to help users scaffold their process of writing a relatable hook for complex scientific topics. We demonstrate that LLMs can help writers find everyday experiences that are relatable and interesting to the public, avoid jargon, and spark curiosity. Our evaluation shows that the system reduces cognitive load and helps people write better hooks. Lastly, we discuss the importance of interactivity with LLMs to preserve the correctness, effectiveness, and authenticity of the writing.

Open science: Preprint

Archived source 1

 Research Article Computer Science
LLM use: Science Communication		 Not recorded
Using AI in Grounded Theory research – a proposed framework for a ChatGPT-based research assistant
Description and source

The purpose of this paper is to explore the potential application of ChatGPT in relation to grounded theory. Our focus is building a case as to its usefulness to support the research process as an assistant to the researcher, rather than to replace the intellectual rigour needed to conduct credible grounded theory research. To aid this, we present a framework for using ChatGPT to assist researchers’ decision making and analysis. By structuring the analytical process into clear research phases - from initial coding through to visualisation and expansion - and providing specific prompts and instructions for each phase, the framework enables researchers to systematically harness AI capabilities whilst maintaining the methodological rigour and accountability of the researcher in leading this process. We argue that the framework's strength lies in its careful alignment with established grounded theory processes, particularly in its emphasis on constant comparison throughout all analytical phases. As many grounded theory methods are employed in other qualitative research designs, we argue that the proposed framework may have potential for use in a broad range of designs, however, we also suggest that this is the start of new conversations about how researchers can harness AI to assist their decision making and intellectual work, processes which can never be fully replaced.

Open science: Preprint

Archived source 1

 Discussion Article Sociology
LLM use: Data Analysis
What Limits LLM-based Human Simulation: LLMs or Our Design?
Description and source

We argue that advancing LLM-based human simulation requires addressing both LLM's inherent limitations and simulation framework design challenges. Recent studies have revealed significant gaps between LLM-based human simulations and real-world observations, highlighting these dual challenges. To address these gaps, we present a comprehensive analysis of LLM limitations and our design issues, proposing targeted solutions for both aspects. Furthermore, we explore future directions that address both challenges simultaneously, particularly in data collection, LLM generation, and evaluation. To support further research in this field, we provide a curated collection of LLM-based human simulation resource

Open science: Preprint

Archived source 1

 Discussion Article Computer Science
LLM use: Data Generation
Why and how to embrace AI such as ChatGPT in your academic life
Description and source

Generative artificial intelligence (AI), including large language models (LLMs), is poised to transform scientific research, enabling researchers to elevate their research productivity. This article presents a how-to guide for employing LLMs in academic settings, focusing on their unique strengths, constraints and implications through the lens of philosophy of science and epistemology. Using ChatGPT as a case study, I identify and elaborate on three attributes contributing to its effectiveness—intelligence, versatility and collaboration—accompanied by tips on crafting effective prompts, practical use cases and a living resource online (https://osf.io/8vpwu/). Next, I evaluate the limitations of generative AI and its implications for ethical use, equality and education. Regarding ethical and responsible use, I argue from technical and epistemic standpoints that there is no need to restrict the scope or nature of AI assistance, provided that its use is transparently disclosed. A pressing challenge, however, lies in detecting fake research, which can be mitigated by embracing open science practices, such as transparent peer review and sharing data, code and materials. Addressing equality, I contend that while generative AI may promote equality for some, it may simultaneously exacerbate disparities for others—an issue with potentially significant yet unclear ramifications as it unfolds. Lastly, I consider the implications for education, advocating for active engagement with LLMs and cultivating students' critical thinking and analytical skills. The how-to guide seeks to empower researchers with the knowledge and resources necessary to effectively harness generative AI while navigating the complex ethical dilemmas intrinsic to its application.

Open science: Preprint

Archived source 1

 Research Article; Documentation; Tutorial w/o Code; Application/Tool; Discussion Article; Use Case Example Any Discipline
LLM use: Research Design; Data Collection; Data Cleaning/Preparation; Data Generation; Dataset Joining; Data Analysis; Describing Results; Web Scraping; Science Communication; Other

Suggest a resource

To suggest an addition or correction, open an issue on GitHub.