Since the release of ChatGPT to the public in 2022, interest in artificial intelligence (AI), machine learning (ML), and specifically large language models (LLMs) has skyrocketed. The healthcare sector has been no exception, with countless use cases being devised and theorized.^1–4 Within otolaryngology-head and neck surgery (OHNS), investigations into the efficacy and utility of LLMs have likewise proliferated rapidly, with reports focused on diagnosis, clinical decision making, writing skills, and test-taking abilities, to name just a few.^5,6

However, additional scholarship has delineated that different LLMs, and even seemingly similar queries to a given LLM, are not equivalent. Firstly, LLMs can be fine-tuned – i.e. additional training data can be incorporated – to conform to a specific task. LLM responses can also be affected by subtle changes in the prompt given to the model; this has given rise to “prompt engineering,” or the design of prompts to optimize desired output.⁷ Many studies have extensively documented the wide variability in LLMs’ ability to answer biomedical and clinical questions based on different prompting strategies.^8–12 A recent investigation additionally recommended running prompts multiple times, as this can uncover inconsistencies in LLM responses that one-time prompting will often miss.¹³

What follows from the reliance of LLMs on prompting is a potential lack of generalizability of LLM-focused studies: two investigations assessing similar clinical abilities of the same LLM could find markedly different results based solely on their prompting strategies. There are also implications for reproducibility – the ability of individuals outside the original research team to independently arrive at similar findings with the same methodology and data.¹⁴ With the recent introduction of reasoning models, prompting recommendations between models now vary significantly.^15,16 This necessitates rigorous, transparent and standardized reporting of methodology in studies querying LLMs. Unfortunately, these practices are not uniform in LLM-focused research at large.⁷ This leads to potential misperceptions of LLMs’ abilities, which, when implications regarding healthcare are raised, can have significant ethical repercussions.¹⁷ Additional ethical concerns – such as patient privacy and algorithmic bias – also necessitate careful approaches within research using LLMs, though these concepts are often not represented in existing publications.⁴

These factors have led to the development of consensus statements recommending reporting criteria, such as TRIPOD-LLM and the DEAL-B checklist.^18,19 These criteria include not only a complete description of the LLM being used, but also details about prompt development and fine-tuning, evaluation of output, use of patient-specific data, and discussion of ethical issues such as bias. However, while overall LLM use within OHNS research has been well-catalogued by recent scoping reviews, these publications have not assessed methodology or methodological reporting.^5,6

Given the known profound impacts of different methodological approaches to LLMs, as well as the uneven compliance with recommendations regarding descriptions of methodology in biomedical research at large, we sought to undertake a scoping review of LLM-focused literature within OHNS, to assess quality and consistency of methodology reporting and to evaluate reproducibility of the findings within LLM research in our specialty.

This review was reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISM-ScR) checklist.²⁰ The review protocol was delineated in advance and made available on October 7, 2024 on Open Science Framework (https://osf.io/pzyec/?view_only=e55078e6f3414545be942ad872adfd7c).

After removal of duplicates, 925 unique studies were captured by the literature search. Full texts were reviewed for 157 publications; 40 were ultimately excluded, yielding 117 studies deemed appropriate for inclusion. Cohen’s Kappa for the abstract and full text screening were 0.84 and 0.78, respectively, corresponding to “near-perfect” and “substantial” agreement. Study selection is summarized via a PRISMA flow diagram in Figure 1. Included studies are listed in Supplemental Table 1, and full citations are included in Appendix 3. Recent reviews have already robustly catalogued the prevalence and primary applications of LLMs within OHNS and other fields, thus this basic information for our review’s included studies is included in Appendix 4 and additionally summarized in Supplemental Figure 1 and Supplemental Figure 2.^6,21,22 The results of the additional search for publications from June 2025 are summarized in Appendix 5 and Supplemental Table 2 and compiled as citations in Appendix 6; the results reported herein reflect only the initial search.

In this scoping review, we have catalogued the methodology used in LLM research within OHNS literature and aimed to describe the quality, consistency and transparency of reporting to assess reproducibility. To the authors’ knowledge, this manuscript represents the first scoping review to assess LLM methodology within OHNS. As the use of LLMs has skyrocketed both in the world at large and in OHNS research, a description of the current state of scholarship in this area is ever more important to ensure quality of research. Furthermore, since LLMs have introduced a previously untapped – and rapidly evolving – avenue for research, many studies have been published in the relative absence of standardized reporting guidelines, necessitating a thorough understanding of the quality of these investigations. Our review found marked variability in completeness of methodological reporting, with the vast majority of studies omitting key factors, especially with regards to description of LLM prompting strategies.

For example, fewer than half of the reviewed publications reported the date of LLM use; this is notable because LLM performance on identical tasks can vary over time.²⁶ In addition, very few studies reported adjusting model parameters or employing fine-tuning techniques to improve LLMs’ performance. Because fine-tuning can have significant impacts on LLMs’ abilities to answer medical questions or accomplish tasks^27–29 – especially in a field such as OHNS with highly specific knowledge – not only is the use of fine-tuning more likely to maximize clinical applicability, but also the absence of specifying whether fine-tuning has taken place leads to significant questions about the generalizability of a study’s results.

Relatedly, although most studies reported the number of prompts used, fewer than half published direct quotations of all prompts. Furthermore, a very small minority provided any rationale for the number of prompts used, such as performing a power calculation or otherwise justifying sample size, explicitly describing resource constraints limiting the number of prompts, or reporting reaching a point of saturation. Perhaps most strikingly, however, was that almost 90% of studies failed to provide a specific rationale for why prompts were written a particular way, and only 6% performed any kind of prompt testing or refinement. It is difficult to overstate the importance of prompting strategies in determining LLM outputs; some studies suggest that prompt engineering alone can be more productive than fine-tuning (which can often be time-consuming and/or expensive) for medical applications of LLMs.^11,30 Even aspects as seemingly trivial as formatting can affect prompt outputs, necessitating the sharing of direct quotes or screenshots of prompts to ensure reproducibility.³¹ In addition, one publication found that careful attention to prompt design and iteration could reduce major LLM hallucinations by 75%.³² Thus, in the absence of precise reporting around the prompts used, the ways in which they were entered, and the decisions surrounding their development, the generalizability of studies employing LLMs within OHNS is thrown into question.

It is important to note that there is no evidence that the quality of reporting is improving over time; for example, of the 86 of our included studies published by June 2024, 43 (50%) included direct quotations of all prompts, compared with 11/31 (35.4%) of studies published since July 2024. The emergence of TRIPOD-LLM and DEAL-B as standardized frameworks for reporting LLM studies not only highlights the importance of publishing this information, but also provides easy-to-implement checklists for investigators carrying out such studies, potentially laying the groundwork for improved reporting in the future, since all included studies from the initial search were published prior to these formal reporting recommendations.^18,19 However, the repeated search gathering studies published in June 2025 failed to show significant improvement; fewer than half of these studies provided direct quotations of all prompts, and only small minorities justified prompt design decisions, performed prompt testing or refinement, or rationalized the number of prompts used.

Several key concepts underpin most prompt engineering strategies. The first is the use of zero-shot, one-shot, or few-shot prompts – namely, giving the LLM zero, one, or multiple examples of an ideal output in the prompt.¹¹ Chain-of-thought prompting is another technique which specifically asks the LLM to perform tasks in a step-by-step fashion or explain its reasoning, while a related strategy known as heuristic prompting includes pre-defined rules in the prompt to aid the LLM in decision-making.⁸ Some comparisons of prompting strategies have found that zero-shot prompting was more likely to be successful for basic tasks, while unfamiliar questions or inputs requiring domain-specific knowledge often required few-shot prompts to maximize efficacy.^9,10 However, other studies have suggested that optimal prompt strategies vary significantly between different LLMs and specific clinical applications.⁸ A potential difficulty with few-shot prompting is the requirement for additional tokens, or units of data used to process text; not only does this increase cost for paid versions of LLMs, but some studies also suggest that increased token counts are associated with worse performance.^10,11

Additional “best-practice” prompt design tools include asking questions as specifically as possible, directly describing the LLM’s role (also known as the “Persona Pattern,”), and testing and refining prompts until desired outputs are achieved.³³ It is important to note that beyond prompt engineering, additional factors – such as utilizing the LLM’s “memory” by asking questions asked in the same session, or adjusting model parameters such as temperature (a measure of randomness or “creativity”) – can also significantly influence LLMs’ outputs.^33,34

It is also important to note that relatively few studies explicitly discussed governance concerns surrounding LLMs, such as algorithmic bias and patient privacy. This is particularly important given that 65% of U.S. hospitals use AI and predictive models, though only 44% evaluate these models for bias.³⁵ In addition, while only a minority of reviewed publications entered patient data into LLMs, some of these studies did not report obtaining informed consent from individuals whose data was used. While evaluating LLMs’ abilities to provide accurate patient-facing information within OHNS has its value, LLMs can also make real provider-facing contributions to clinical workflows, such as extraction and summarization of electronic health record data,³⁶ writing notes,³⁷ and providing clinical decision-making support.³⁸ However, robust studies have demonstrated persistent biases against minoritized or disadvantaged groups (such as Black, unhoused or LGBTQIA+ individuals) when LLMs are asked to provide healthcare recommendations.³⁹ It is thus crucial that investigators of LLMs within OHNS carefully consider concerns around ethics and privacy in order to advance our field and leverage the capabilities of LLMs to better serve patients.

Despite the important contributions of this review, there are relevant limitations to discuss. Firstly, given the nature of this manuscript as a scoping review, analysis of the results of the included studies is not within our purview, and we can thus not make any summative claims about the overall efficacy of LLMs in achieving the tasks set by OHNS researchers. In addition, certain classifications in our data extraction cannot be universally applied to all LLM studies; for example, LLM pipelines that use chain-of-thought techniques cannot be easily described in terms of the number of runs per prompt. Furthermore, as previously noted, the rate of publication – and of change – within the field of LLM research is incredibly rapid; thus, recently published studies from November 2024 to the present are not included in this review, limiting our ability to comment on potential improvements over time in methodological reporting quality. Importantly, despite the fact that prior standards for reporting in AI research predate the vast majority of LLM-focused OHNS research,⁴⁰ the most up-to-date recommendations regarding methodology reporting in LLM research – such as the aforementioned TRIPOD-LLM¹⁸ and DEAL-B¹⁹ – have been released after the publication of the studies included in the initial search for this review, and thus it could be supposed that the quality of reporting in our field has improved in recent months. We sought to address these limitations by performing an additional search cataloguing publications from June 2025, but we acknowledge that this provides only a snapshot of potential evolution in manuscript reporting. It is also important to note that despite basic comparisons to assess potential changes in quality of reporting over time, our methodology does not allow for more in-depth or definitive longitudinal analysis, and future studies could seek to more robustly address whether methodology reporting is improving in our field.

It is the authors’ hope that illuminating the variability and shortcomings of methodology reporting in LLM research in our field can serve as a call to action for investigators and journals alike to strive for consistent, uniform, and reliable reporting in future studies, allowing for a higher-quality and more clearly generalizable evidence base.

In this scoping review, we have provided the first robust catalogue of LLM methodology in OHNS with a focus on reproducibility. Reporting of methodology in accessing and prompting LLMs is variable and frequently poor, leading to substantial questions regarding the generalizability of the vast majority of the reviewed studies. Careful attention by researchers and journals within OHNS towards higher standards in methodology reporting will be crucial to ensure reproducibility, strengthen the evidence base surrounding LLMs in OHNS and to eventually leverage these tools for patient benefit.