In defining the requirements of the AI Copilot a team of two experienced otolaryngologists (NB and AR) determined that when performed on a simplified model (AirSim Combo Bronchi X mannikin [Figure 1] (United Kingdom, TruCorp Ltd) a basic FFL procedure consisted of:

Entering the nasal cavity and navigating the nasal passage to reach the nasopharynx

A computer scientist then translated these high-level requirements to specific capabilities the AI Copilot would need:

Identifying the optimal scoping path in the nasal passage

To develop the AI Copilot, we used supervised machine learning, in which neural networks learn to predict output labels from human labeled data. The AI copilot consisted of two key machine learning components:

An image classifier model, dubbed the “anatomical region classifier”, responsible for predicting the location of camera in the upper airway.

The outputs of these models were then filtered and at times averaged to reduce noise in the system. These outputs were then incorporated as inputs into the logic of a larger system that kept track of the state of the procedure and the camera location. Based on these inputs and the state of the system, instructions and cues were provided to the user via an overlay on the video feed.

To run the AI Copilot, the live video feed from an Ambu aView 2 Advance was transferred to a computer via an HDMI connection and a video capture card. Running a local FastAPI web server¹⁷, the computer read in the video feed, processed it, and sent the modified feed along with metadata to a web browser through a webhook. This new video feed was displayed on the computer monitor, allowing the Machine Learning Copilot to provide real-time cues to the user overlayed on top of the raw FFL video. See Video 1 for a representation of the cues provided.

The training data was collected by performing FFL on an AirSim Combo Bronchi X mannikin [Figure 1] (United Kingdom, TruCorp Ltd) using an Ambu aScope 4 RhinoLaryngo Slim [Figure 1] connected to an AmbuÒ Viewä 2 Advance Displaying Unit (Ballerup, Ambu A/S) [Figure 1]. A representative video of the AI copilot annotating the FFL in real time can be seen in Video 1.

Data for training both models were selected from a dataset that was generated by performing 20 flexible laryngoscopies on the mannikin. All scoping videos were performed by a single, experienced otolaryngologist on an Ambu aScope 4 RhinoLaryngo Slim [Figure 1]. Various levels of expertise were simulated when recording videos to ensure a variety of angles and views were captured. The FFL AI copilot was trained on the left nasal cavity only to keep the machine learning consistent, and so there was no confusion or misclassification of structures based on sidedness.

The anatomical region classifier was used to identify the physical location of the camera at the time of image capture. The classes used were Larynx, Nasal Cavity, Nasopharynx, Oropharynx, Out of Body, and Whiteout. Videos were split into image frames and each image was assigned to a class by a trained graduate student. The classes were defined in the manikin model as follows:

Out of body: Any image from outside of the mannikin.

In total there were 56,262 images in the anatomical region dataset. They were broken down into 42,971 training images, 3,596 validations images, and 9,695 testing images. See Table 1 for breakdown of the class splits. To prevent data leakage due to the high correlation between consecutive frames, each video and all its images were assigned exclusively to a single split: train, validation, or test.

The region classifier used the Resnet18 convolutional neural network model pretrained on ImageNet and was finetuned on the new dataset designed for anatomical region classification within the mannikin¹⁸. The final linear layer of the Resnet18 was replaced with a new linear layer with six outputs for the six anatomical regions. Models were trained using a sweep of hyperparameters of frozen layers, learning rate, image transforms, and batch size.

The anatomical structure object detector is used to identify key anatomical structures in the mannikin and place a bounding box around each structure. The anatomical structures labeled were Inferior turbinate, Middle turbinate, Uvula, Vallecula, Epiglottis and Vocal Folds. In addition, the path we desired for the user to take in the nasal cavity was labeled into 2 separate classes, one for the path leading up to the middle turbinate (Path 1), and one for the path after passing the proximal end of the inferior turbinate (Path 2). Images were labeled with bounding boxes by trained graduate students using Label Studio, a data annotation tool¹⁹. Some of the structures did not have apparent boundaries with clear delineations, which led to noisy bounding box labels. A confusion matrix was generated for the anatomical region classifier which can be seen in [Figure 2].

The anatomical structure detector used a YoloV7 model that was finetuned on a dataset made of 11,337 images from a subset of 16 videos and evaluated against 3,096 images from 4 videos²⁰. No validation set was used due to the time intensive nature of labeling the data. See Table 2 for breakdown of the class splits. Models were trained using a hyperparameter sweep of image size, learning rate, image transforms, and batch size. Since multiple structures could be identified within a single frame, predicting the maximum likelihood class was not viable when integrating the anatomical structure detector into the AI Copilot. Instead, each class had its own confidence threshold that was hand-tuned based on human judgement from interacting with the complete system.

The performance of the model was measured using mean average precision (mAP). Mean average precision is a standard metric used in object detection and has the benefit of balancing precision and recall. An intersection over union threshold of 0.5 was used to calculate the mAP since the ground truth bounding box labels for some classes were noisy.

The AI copilot was pilot tested prospectively by having 64 medical students naïve to FFL use the AI copilot to perform FFL on the AirSim Combo Bronchi X mannikin [Figure 1] (United Kingdom, TruCorp Ltd). After using the FFL, anonymous surveys were handed out to the medical students, and they were then asked to rate the ease of use of machine learning copilot during FFL, as well as self-rate their FFL skills with and without the Copilot, both on a 5-point Likert Scale. Figure 3 shows percentages of students who answered within each Likert category (strongly disagree, somewhat disagree, neutral, somewhat agree, strongly agree). Descriptive statistics were used to analyze medical student responses to ease of use of the tool, and subjective skill set before and after use of the tool. This was a proof-of-concept study to test the feasibility of the AI copilot. The authors plan to do a formal study evaluating the impact of the AI copilot on novice learners.

The best performing model had 6 frozen layers and was trained with a batch size of 128 for 4 epochs with a learning rate of .0002. This model achieved an overall accuracy of 91.9% on the validation set and 80.1% on the test set. A breakdown of the per class metrics can be found in Table 3. From the confusion matrix [Figure 2], it was apparent that the whiteout class was most frequently incorrectly classified as the nasal cavity where most of the whiteouts happened in the training data. Anatomical regions were most likely to be misclassified as one of their adjacent regions since the transition point from one region to another is not discrete.

The best performing model was trained for 80 epochs using a batch size of 128, and a learning rate of .001. This model achieved an overall mean average precision of 0.642. A breakdown of the per class metrics can be found in Table 4. Large variability was seen in the mAP across classes. With strong delineations, structures like the vocal folds and epiglottis likely had more consistency and less noise in their bounding box labeling, and therefore were easier for the model to learn.

Initially, the AI Copilot was designed to run on a computer with a graphics processing unit (GPU), but we felt this limited the potential environments in which this might eventually be used. To make the software more broadly usable we focused on making it work on a MacBook Pro M1. Through various optimizations we were able to run the AI Copilot at approximately 28 frames per second (FPS), which is imperceptible from real time and nearly matches the video frame rate of 30 FPS.

As seen in Figure 3, 90.9% of medical students naïve to FFL strongly agreed/agreed that the AI copilot was easy to use. However, as illustrated on Figure 4, medical students’ self-rating of FFL skills following use of AI copilot were overall equivocal to their self-rating without the co-pilot.

We described the development and pilot testing of the first AI copilot to help train novices to competently perform FFL on a mannikin. The AI copilot tracked successful capture of diagnosable views of key anatomical structures effectively guiding users through FFL to ensure all anatomical structures are sufficiently captured. This tool has the potential to assist novices in efficiently gaining competence in FFL.