Viral hepatitis represents a significant public health challenge. In 2017, the National Academies of Science, Engineering, and Medicine released a strategy for the elimination of hepatitis B and C viruses (HBV, HCV) from the United States. Associated recommendations included working with states to improve viral hepatitis screening and information collection (Buckley and Strom, 2017).

The Global Hepatitis Outbreak and Surveillance Technology (GHOST) system is a program of the Centers for Disease Control and Prevention (CDC) to address the enhanced hepatitis screening recommendation. GHOST is an epidemiological decision support and molecular data collection system. The system clusters related cases, allowing the source tracing of hepatitis outbreaks. It also highlights unusual situations useful for epidemiological investigations, such as superinfections of HCV, which are transient phenomena that are only regularly observable in high-risk environments such as injection drug use. GHOST uses a deep-amplicon-based sequencing protocol that targets a region of the HCV genome called hypervariable region 1 (HVR1; Longmire et al., 2017). The sequencing protocol is intended to sample the structure of the intra-host HCV population found in a sample. To do so, we have developed an error-correction and filtering pipeline that uses a 2 × 300 bp sequencing kit on an amplicon with a design length of 264. Therefore, many bases of each molecule are sequenced twice, once in the first read and once in the second read.

As part of each cycle, the sequencing machine estimates a confidence value known as the PHRED score, which is output along with the base call. Originally developed for Sanger Sequencing, the PHRED score is defined in equation 1 where p is the estimated probability that a base was called incorrectly (Ewing and Green, 1998) (Ewing et al., 1998).

For such a binary confidence value to be calibrated, the events that are assigned a probability of 30% should have a long-run error frequency of 30% (Dawid, 1982). The most direct method of evaluating the calibration of the PHRED scores involves measuring substitution errors made in a known sequence. This has been used by Schirmer et al. (2015) to measure the calibration of sequencing amplicons on the MiSeq instrument. These experiments are useful in characterizing the MiSeq instrument, but they rely on a known ground truth of the sequences. Such experiments are not commonly performed because they do not produce other information. Examination of the overlap of paired end sequences has been used to evaluate the overall usefulness of quality scores by Eren et al. (2013). Their article developed an error filtering method based on overlapping reads matching across the entire overlap length. However, they did not investigate the quality scores themselves, instead examining how quality score-based read filtering methods changed the fraction of reads that merged using their criteria. Zhang et al. (2017) developed a method for estimating the PHRED values from raw data generated during HiSeq operation. Their method used runs of phiX174 sequences as known sequences. Illumina MiSeq uses PhiX174 to increase the complexity of the sequences on the chip to improve cluster detection in the raw image data captured from the flow cell. Errors were identified using a short-read alignment to the phiX174 genome. They compared base calls with the known phiX174 sequence, detected errors, and subsequently evaluated their new scoring method with the Illumina HiSeq built-in method. The Zhang et al. method used several features from the raw data used by the on-instrument base calling process, which is not available in our datasets, as we only receive the base calling results.

As GHOST reads purposefully overlap, every sample sequenced by the GHOST protocol provides a natural indirect experiment in the calibration of the PHRED scores. Analyzing the intentional overlap of paired reads allows for the isolation of sequencing errors from errors incurred during library construction. This is a significant advantage over existing methods that conflate the two error sources. GHOST is accepting sequences from multiple MiSeq instruments and has accumulated many runs using the overlapping protocol. These datasets provide an opportunity to measure the distributions of many features of the performance of the Illumina MiSeq instrument under varying conditions.

As an automated system making important public health recommendations, GHOST must be able to determine the quality of the data that motivates these recommendations. The authors have published two articles describing the specifics of sequencing quality monitoring Longmire et al. (2017) and Sims et al. (2018). At multiple points along the quality control pipeline, a sample can be rejected, and a suggestion is returned to the user on what likely caused the rejection and how to improve the sequencing quality. The reads are subjected to steps such as verifying sequence identifiers, verifying the expected primers, merging reads, checking for open reading frames, and a genotyping step, which rejects samples that cannot be assigned clearly to a subtype.

In this study, we intend to assess how accurate the MiSeq’s internal error model, its calibration, and its usefulness for discrimination of sequencing errors using observations of overlapped paired-end reads. A maximum likelihood (ML) method of estimating the long-run error rate of sequencing errors indexed by machine-assigned PHRED scores is developed. We find that the PHRED scores are not calibrated, but the size of the error is small in most cases. However, the MiSeq error model does not separate sequencing errors from adjacent correct base calls, even though the assigned PHRED scores reflect the long-term error rate of the base call.

We find that the PHRED scores produced are not calibrated, but the absolute size of the error is small. Our MLE calibration method successfully corrected the error as much as possible using the features examined. To improve calibration further, additional features such as per-nucleotide biases or raw intensities would need to be utilized. However, the MLE model does not always improve the calibration, but this can be detected by examining the change in the Brier score achieved.

Although quality scores are accurate to their definition, their resolution (RES) scores are small relative to the total Brier score. The quality scores assigned are relatively accurate in predicting the error rate of the base calls. The base call procedure cannot separate correct calls from substitution errors. It tends to assign substitution errors of similar quality to the surrounding correct base calls. Given the low substitution rate of the MiSeq instrument, this is a relatively calibrated methodology, but it means that the PHRED scores are uninformative of sequencing errors.

The MiSeq instruments use two different light colors with four filters to perform base calling. Base calls generated by the same laser channel are significantly more likely to be substituted for each other than base calls generated by different channels. We note that in 98.90%9091 of the datasets have on-channel transitions greater than would be expected if all transitions were equally likely (see Fig. 6). This also mirrors previous results (Schirmer et al., 2015).

Most of the findings reported in this article were foreshadowed by Schirmer et al. (2015), Eren et al. (2013), and Zhang et al., (2017). Eren et al. (2013) addressed quality scores’ effects on downstream analysis and not the quality scores themselves. They found that even reads with consistently high-quality scores contained many mismatched base calls in their overlap. Schirmer et al. (2015) and Zhang et al. (2017) analyzed the quality scores directly. No direct comparison was made between our method and the method of Zhang et al. because it uses raw instrument intensities, which are not provided to the GHOST system. Schermer et al. use mock communities of known sequences to measure errors, as our dataset does not contain known sequences, we also did not do a direct comparison to this method. These methods could not separate errors in library preparation from sequencing errors. The greatest strength of our analysis is its ability to separate these two sources of error, leading to subtly different results. We found greater agreement between the measured PHRED scores and their true values than those found by Schirmer et al. (2015). Contrary to their results, we found that the high-quality scores seemed to underestimate the true quality of the base calls. We attribute the errors they measured to library preparation. We agree with both Schirmer et al. (2015) and Eren et al. (2013) that PHRED scores are of limited value in identifying substitution errors.

The MLE model is capable of removing the majority of calibration errors from quality scores. In two runs, the prediction error was increased by the MLE estimator. We believe that this is due to the optimization function getting stuck in a local maximum of the likelihood function.

Recalculating the Brier scores with the newly calibrated values produced an average relative improvement of 2.83%. The maximum improvement that a run-specific calibration curve can achieve is to reduce the reliability measure to zero. This was achieved in most cases. However, prediction accuracy can only be improved by considering features beyond PHRED scores.

As part of the current quality control process, the CASPER method is used to merge the overlapping paired reads (Kwon et al., 2014). By collecting statistics during this step, the calibration of PHRED scores can be included with only minimal extra processing time of less than a second per sample. A new haplotyping method using calibrated PHRED scores is being developed to be published at a later date. The accuracy of PHRED score calibration is important to improve the calling of rare variants, to which GHOST clustering is very sensitive. Currently, GHOST employs a simple method of excluding low-reliability haplotype calls; haplotypes observed fewer than 10 times are excluded. Using the calibrated PHRED scores, we can reach more deeply into the low-frequency tail of haplotypes to include rarer haplotypes in our analysis.

We also intend to further develop the methods presented in this article to provide additional quality monitoring of the sequencing of GHOST submissions (Sims et al., 2018). The new quality monitoring will include new quality control thresholds and suggestions for the laboratory process. GHOST currently requires a variant to be seen 10 times or more in a dataset to be used in clustering.

PHRED scores are generally accurate on the basis of their ability to predict matching. Although the Spiegelhalter z-statistic showed that the scores were not calibrated (p<0.0001), the absolute deviations from the expected values were small. However, the calibration of the scores varied significantly among runs. Generally, the raw PHRED scores accurately predicted the probability of a match, and our methods improved their calibration. Even when recalibrated, the PHRED scores were not very informative of sequencing errors, as the MiSeq instrument does not separate correct base calls from substitution errors and generally assigns the background error frequency of the cycle. Given the very low error rate of the MiSeq, this tends to correctly assign high-quality scores to most calls.