Design Considerations

As a starting place, here are initial recommendations of questions to be asked when considering the use of an LLM as part of your scientific research workflow.

  • Where in your research workflow do you plan to use an LLM (e.g., literature review, data collection, data preparation, data analysis, writing)?
  • Based on the research tasks for the LLM, which LLM (e.g., ChatGPT, BARD, Claude) is most appropriate? What characteristics weigh into this decision?
  • How will the LLM complement and/or supplement other research tasks in workflow?
  • Are you planning to use a website LLM (e.g., ChatGPT) or use an API to connect to the LLM?
  • What completion parameters do you plan to manipulate when using the LLM?
  • How will LLM responses be evaluated for accuracy, bias, and other potential limitations?
  • How will you provide for data security and privacy when using the LLM?
  • What methods will you apply to mitigate the risk of inaccuracies, biases, and/or plagiarism in LLM-generated results?
  • How will you comply with applicable institutional and/or regulatory guidelines for using LLMs in research?
  • Will you pre-register the study?