A number of Clinical Laboratory Improvements Act (CLIA)-required performance parameters were adopted, with modifications for the validation of WGS (Table 1). The modular validation template and a summary are presented in Fig. 1. The quality assurance (QA) and quality control (QC) measures are described in detail later in Results.

The accuracy of WGS was divided into three components: platform accuracy, assay accuracy, and bioinformatics pipeline accuracy.

Platform accuracy was assessed as the accuracy of the identification of individual base pairs (“base calling”) in the bacterial genome. The accuracy of the platform was established by determining the proximity of agreement between base callings made by the MiSeq sequencer (measured value) and the NCBI/CDC reference sequence (the true value). We determined the accuracy of the MiSeq Illumina platform by mapping generated reads to the corresponding reference sequence and identifying single nucleotide polymorphisms (SNPs) throughout the genome. A few validation samples differed from the reference genome by several SNPs. However, 99% (324 out of 327) of these SNPs were reproducible among all five replicates sequenced for each sample. Since amplification and sequencing errors were random between different library preparations, it was unlikely that the same erroneous SNP would occur in all five replicates. Therefore, we concluded that these discrepancies were not caused by sequencing errors but most likely were the result of the accumulation of mutations in the reference strains or previous sequencing mistakes in the reference sequence. Sanger sequencing confirmed the SNPs found in WGS results for the selected isolates (see Appendix 14 in Document S1 in the supplemental material). However, in several instances, it was impossible to design specific primers within reach of Sanger sequencing read lengths because SNPs were found in the repeat regions. In one case, the results of Sanger sequencing matched the results of mapping, but upon comparison with de novo assemblies, the same sequence was found in two variants, with and without the SNP detected by mapping. This highlighted the limitation of the short-read sequencing technology and the underlying caution that must be exercised in the interpretation of the SNPs called in the repeat regions of the genomes. In both cases, whether we took into account all SNPs detected between validation and reference sequences or only those SNPs that did not appear in all of the replicates (true sequencing errors), we observed >99.999% agreement of the generated whole-genome sequences with the reference sequences for each tested sample.

Assay accuracy was defined as the agreement of the assay results for the validation sequences with the assay results for the reference sequences of the same strains. Four applications of WGS were used to validate the accuracy of the assay: an in silico multilocus sequence typing (MLST) assay, a 16S rRNA gene species identification assay, an assay for the detection of ABR genes, and a genotyping assay using high-quality SNPs (hqSNPs).

The definition for a correct result for MLST corresponds to the correct identification of each of the MLST alleles in the validation sequence as well as in the reference sequence analyzed in parallel. For all validation samples, each of the sequences of the seven housekeeping genes used in the typing scheme (or six genes for Aeromonas hydrophila) were identified correctly, resulting in 100% allele identification accuracy.

For ABR gene detection, the sequences of the ATCC susceptibility control strains generated during validation were analyzed by using ResFinder. Data analysis using ResFinder implies comparison of the sequence against each entry in the ResFinder database, which at the moment of validation contained sequences of 1,719 antibiotic resistance genes, resulting in a total of 1,719 tests performed for each validation sample. The results of ResFinder detection were compared to the resistance genes known to be present in the ATCC strains. Two of the sequenced isolates contained one resistance gene each, which were also detected by ResFinder, with no additional genes being identified. No resistance genes were detected by ResFinder in the samples suggested to be negative susceptibility controls and lacking resistance determinants. Thus, the accuracy of the assay for ABR gene detection was 100% (see the section on detection of resistance genes in ATCC strains using ResFinder in Document S1 in the supplemental material). Moreover, 13 reference sequences of Gram-negative (7 sequences) and Gram-positive (6 sequences) bacterial isolates with various resistance genes from the U.S. Food and Drug Administration (FDA)-CDC Antimicrobial Resistance Isolate (AR) Bank were used for in silico testing of ABR gene detection accuracy using ResFinder. The resistance genes in the AR Isolate Bank isolates were previously detected by the CDC using PCR-based methods (for primary resistance types) and by ResFinder (database last updated 2 June 2016). Our analysis of reference sequences with ResFinder (last updated 17 February 2017) confirmed the presence of all genes that were detected by PCR-based methods (n = 8), resulting in 100% accuracy. At the moment of analysis, ResFinder did not have the ability to detect the truncation of porin genes; therefore, porin-related resistance mechanisms mentioned in the CDC database could not be detected with ResFinder. Upon ResFinder analysis at the CDC, the isolates harbored a total of 83 resistance genes (representative of 57 different alleles). The detection of the genes by the CDC using ResFinder was replicated in this study, with few discrepancies (see the section on in silico detection of resistance genes in FDA-CDC isolates using ResFinder and Appendix 15 in Document S1 in the supplemental material). In the case of Gram-negative bacteria, all discrepancies were represented by additional genes detected in comparison with CDC ResFinder results, and these discrepancies were most likely caused by the database update; e.g., the aph(3′)-Ia gene was not on the list of the genes detected with ResFinder by the CDC but was detected in the same sequences using ResFinder in this study. This particular gene discrepancy could be explained by the use of a later version of ResFinder, which was updated with additional aminoglycoside resistance genes. In the case of Gram-positive bacteria, several discrepancies were also caused by the additional genes detected in this study, but two genes that were present in the CDC results were missing from our results. These false-negative genes were detected by us but possessed <99% identity or an incomplete sequence and therefore were excluded from the final result. Nevertheless, the agreement between CDC ResFinder results and our ResFinder results was 99.97%.

For the 16S rRNA identification assay, variations in one gene were detected, so the species identification results as a whole (e.g., “Escherichia coli”) were considered a single test. The identity of the 16S rRNA sequences extracted from validation samples showed 100% matches with the 16S rRNA sequences extracted from the reference sequence.

To assess the accuracy of the genome-wide hqSNP-based genotyping assay, phylogenetic trees were built by using reference sequences and validation sequences, and the resulting trees were compared. For better comparison, we used at least five strains of the same species in the phylogenetic tree. The accuracy of the genotyping assay was determined by using two approaches: (i) topological similarity between the reference tree and the validation tree using Compare2Trees software and (ii) comparison of the clustering patterns of the validation tree and the reference tree. Therefore, in addition to the detection of SNP differences between the validation sequences and reference sequences, their effect on the final tree topology was also estimated to determine the accuracy of the genotyping assay. Phylogenetic trees were generated for five bacterial species (Escherichia coli, Salmonella enterica, Staphylococcus aureus, Enterococcus faecalis, and Stenotrophomonas maltophilia). All five validation trees had matching clustering patterns and 100% topological similarity with the corresponding reference trees (see Table S2 in the supplemental material).

The accuracy of the bioinformatics pipeline for hqSNP-based genotyping by itself was assessed by the recapitulation of previously reported results using WGS raw reads of bacterial isolates included in two previous studies (Table 2). This was an additional assessment of the accuracy of the genotyping pipeline apart from the accuracy of base calling from in-house sequencing of the isolates. Phylogenetic analyses of isolates associated with outbreaks caused by a Gram-positive pathogen in the first study (MRSA [methicillin-resistant Staphylococcus aureus] study) (38) and a Gram-negative pathogen in the second study (Salmonella study) (39) were performed for the validation of the bioinformatics pipeline (Fig. 2). The clustering of the validation tree completely replicated the clustering of the tree from the MRSA study (38) (Fig. 2A and B); e.g., isolates 4 and 5 were identical and clustered together according to the MRSA study, and the same results were shown in the validation tree, with isolates 4 and 5 sharing the same node. All conclusions with regard to the genetic relatedness of the isolates that can be drawn from the tree reported in the MRSA study could also be made for the analysis of the corresponding validation tree. The group of outbreak isolates from the MRSA study was compared with epidemiologically unrelated isolates suggested by the same study (no tree was available from that report). Phylogenetic analysis using the in-house bioinformatics pipeline showed that epidemiologically unrelated isolates did not cluster with the group of outbreak isolates and appeared to be genetically distant (Fig. 2C). Thus, the resulting phylogenetic tree produced by our bioinformatics pipeline showed complete concordance with the epidemiological data.

From the Salmonella study (39), we selected nine isolates that were representative of four independent outbreaks and two isolates that were epidemiologically unrelated controls (Fig. 2D). The clustering of the validation tree was identical to the clustering of the tree from the Salmonella study. For example, isolates 6 and 7 were part of the same outbreak, while isolate 8 was an epidemiologically unrelated control used in that study. In agreement with the epidemiological data and the tree from the Salmonella study, the validation tree showed that isolates 6 and 7 clustered together but not with isolate 8 (Fig. 2E). All observations about the genetic relatedness of the isolates drawn from the tree from the Salmonella study could be replicated from the analysis of the validation tree. In summary, based on in silico analysis of data from both studies, 100% accuracy of our bioinformatics pipeline was established for phylogenetic analysis.

Repeatability (within-run precision) was established as the concordance of the assay results and quality metrics obtained for a sample tested multiple times within the same sequencing run. Reproducibility (between-run precision) was assessed as the consistency of the assay results and quality metrics for the same sample sequenced on different occasions. Thirty-four validation samples each were sequenced three times in the same sequencing run (for repeatability) and three times in different runs (for reproducibility). For within-run replicates, one DNA extract was used, but independent library preparations were done, with the final samples being included in a single sequencing run. Therefore, for each sample, the numbers of within-run replicates and between-run replicates were 3 each, and the total numbers of repeated results were 5. All quality parameters (depth of coverage, uniformity of coverage, and accuracy of base calling [Q score], etc.) did not change significantly for within- and between-run replicates, as determined by a two-tailed t test. For the quality values for all sequenced samples, see the section on inter- and intra-assay agreement and Appendix 8 in Document S1 in the supplemental material.

The reproducibility and repeatability of the WGS assay were evaluated with two methods: evaluation of base calling reproducibility and repeatability per replicate and evaluation of base calling reproducibility and repeatability relative to genome size. All validation samples except C50 yielded identical whole-genome sequences for all three within-run replicates. One out of three within-run replicates for isolate C50 (Pseudomonas aeruginosa ATCC 27853) had one SNP difference from other within-run replicates (Table S3). The repeatability per replicate was 99.02%. Three validation samples had one out of three between-run replicates that differed from the other two between-run replicates. Sample C47 (Staphylococcus epidermidis ATCC 12228) had one between-run replicate with two SNPs that differed from the other replicates. Samples C49 (Streptococcus pneumoniae ATCC 6305) and C55 (Escherichia coli ATCC 25922) each had one between-run replicate that differed from other replicated sequences by one SNP. The reproducibility of base calling per replicate was 97.05%. Both the reproducibility and repeatability of base calling relative to the genome size (in relation to the total number of base pairs of the covered genome size) were >99.9999%.

We also estimated the reproducibility and repeatability of MLST and 16S rRNA identification assays. For MLST, a total of 441 alleles were analyzed for within- or between-run replicates. Each single allele in all validation samples was identified consistently among within- and between-run replicates. Within- and between-run replicates had repeatable/reproducible sequences of 16S rRNA genes and resulted in the consistent identification of the species. The reproducibility and repeatability of allele detection and species identification within and between runs were 100%.

The sensitivity of WGS was assessed as (i) analytical sensitivity (minimum coverage that allows accurate SNP detection) and (ii) diagnostic sensitivity (the likelihood that a WGS assay will detect sequence variation when it is present) (this value reflects the false-negative rate of the assay).

The specificity of WGS was determined as (i) analytical specificity (the ability of an assay to detect only the intended target in the presence of potentially cross-reacting nucleotide sequences) and (ii) diagnostic specificity (the probability that a WGS assay will not detect sequence variations when none are present) (this value reflects the false-positive rate of the assay).

The limit of detection (LOD) is traditionally defined as “the lowest actual concentration of an analyte in a specimen that can be consistently detected … with acceptable precision” (40). The LOD in this sense is not applicable to WGS, which utilizes pure bacterial cultures as starting material; the amount of the analyte (genomic DNA) that is added to the reaction mixture is strictly standardized, and the DNA concentration of each sample is measured by a fluorometric method before each assay. In our workflow, we established the minimum amount of the starting DNA input to be 1 ng (at a concentration of 0.2 ng/μl) and did not process DNA extracts with concentrations of <1 ng/μl. Instead, we determined the LOD of SNP detection (LOD_SNP) by establishing the lowest coverage that allows accurate SNP calling. The LOD_SNP was estimated by modeling different mapping coverages and estimating the number of SNPs called at each of the coverage values. Nine samples representative of different species were in silico downsampled to coverages of 60×, 50×, 40×, 30×, 20×, 15×, 10×, and 5×. The LOD_SNP was established to be 60×, as it was the lowest coverage which yielded accurate SNP detection for all of the samples (see Table S4 and the section on analytical sensitivity of SNP detection in Document S1 in the supplemental material).

Analytical specificity is referred to as the “ability of an assay to detect only the intended target and that quantification of the target is not affected by cross-reactivity from related or potentially interfering nucleic acids or specimen-related conditions” (40). Since our WGS pipeline is not intended for clinical specimens directly, the conditions interfering with fragmentation, amplification, and sequencing processes are minimized and are monitored via multiple QC steps, including DNA and library purity and concentration measurements. However, interference from contaminating nucleotide sequences is much more consequential. Analytical specificity (interference) was determined by creating sequencing files containing a mixture of the reads from two different samples in silico, thus mimicking contamination and demonstrating its effect on mapping metrics (percentage of reads mapped, percentage of the reference sequence covered, etc.) and SNP detection (see Table S5 and the section on the analytical specificity of SNP detection in Document S1 in the supplemental material). As expected, contaminating reads led to a decrease in the percentage of mapped reads and an increase in the portion of unmapped reads. The percentage of reads in pairs decreased for samples containing contaminating reads. Modeled contamination with E. coli C1 and M. tuberculosis C57 sequencing reads did not cause any change in called SNPs. Contamination with any of the other reads led to additional SNPs being called both between the compared samples and with the reference sequence. In the sample contaminated with S. enterica C75, in addition to nonspecific SNPs, one of the SNPs detected previously was missing. The bioinformatics pipeline had a certain tolerance of contaminating reads depending on the nature of the contamination.

Diagnostic sensitivity and specificity were estimated for genotyping and MLST assays.

To estimate the diagnostic sensitivity and specificity of WGS-based genotyping, the hqSNP phylogenetic trees generated from the validation sequences were compared to the trees generated from the reference sequences for the same strain. In the case of whole-genome sequence genotyping, the true sequence variation (SNP), which was not detected, represents a false-negative result. Changes introduced into the DNA sequence during library preparation/sequencing or data analysis errors could result in false-positive sequence variations. All generated validation trees repeated clustering and had 100% topological similarity to the corresponding reference trees, indicating the absence of false-negative or false-positive results of the genotyping assay. The hqSNP-based genotyping assay was 100% sensitive and specific.

The sequence types of validation sequences and reference sequences were determined by using organism-specific MLST databases. For MLST, the number of true-positive results corresponds to the number of alleles correctly identified in the validation samples. For true-negative results, we performed a comparison of validation sequences against MLST databases for unmatched species, e.g., a search of alleles for the Escherichia coli C1 validation sample against the MLST database for Salmonella enterica. In the latter case, the MLST assay was expected not to identify any alleles. All alleles in the positive validation samples were identified correctly. None of the alleles were identified in the negative controls. Both the diagnostic sensitivity and analytical specificity of the in silico MLST assay were 100%.

The following information about the sequenced genome was collected for the reportable range: genome-wide hqSNPs, housekeeping genes used in MLST schemes, 16S rRNA genes, and antibiotic resistance genes included in the ResFinder database. A reporting language was developed to assist in the interpretation of results by an end user with or without specific WGS knowledge. The appropriate template and examples are provided in Document S1 in the supplemental material.

QA and QC measures were developed to ensure high quality and consistency of routine testing using the MiSeq Illumina platform. QC was performed during the preanalytical (DNA isolation and library preparation), analytical (quality metrics of the sequencing run), and postanalytical (data analysis) steps of WGS. Five QC checkpoints were implemented throughout the WGS procedure: DNA template QC, library QC, sequencing run QC, raw data QC, and data analysis QC. The quality parameters evaluated at each of these QC checkpoints are summarized in Fig. 3. The QA and QC manual established for WGS applications is presented in Document S1 in the supplemental material. Based on the preliminary quality thresholds, all runs passed from the first attempt, and none of the samples had to be resequenced. The final quality thresholds were set at the lower border of quality values, which still allowed the generation of accurate and reproducible assay results (phylogenetic analysis, MLST, 16S identification, or ABR detection). The WGS quality cutoff values are summarized in Table S6 in the supplemental material. We determined the optimal depth of coverage to be ≥60× based on the accuracy of SNP detection at various simulated genome coverages. Table S6 does not reflect the optimal data quality parameters but merely the thresholds below which data will be rejected and the sample will have to be resequenced.

We employed two levels of positive and negative controls. First, as an internal positive control, we used spiked-in PhiX Control (Illumina Inc., San Diego, CA, USA) and evaluated its sequencing error rate. As an internal negative control, at the demultiplexing step, we called index combinations, which did not correspond to any samples in the current sequencing run but were used in the previous run. This negative-index-combination control allowed us to capture carryover contamination with the library fragments generated in the previous runs. Second, as an external positive control, we included E. coli strain ATCC 25922 processed from the DNA extraction step all the way to the data analysis step. As an external negative control, a no-template control was processed through all WGS steps starting with DNA extraction. Additional positive controls representing one Gram-positive strain and one M. tuberculosis strain were introduced solely for the DNA extraction step in cases when the corresponding types of samples were processed (to control for differences in the DNA extraction protocols). The quality cutoff values for the controls can be found in Table S6.

Analytical sensitivity (LOD of SNP detection) was established at a 60× depth of genome coverage. Analytical specificity in the presence of interfering sequencing reads was demonstrated for different types of contamination. The WGS assay was shown to have >99.9% accuracy, >99.9% reproducibility/repeatability, and 100% diagnostic specificity and sensitivity. These parameters met the CLIA requirements for laboratory-developed tests (LDTs).

A set of 34 bacterial isolates representing the typical workflow in PHLs was used for validation and quality control of WGS. These isolates included 10 Enterobacteriaceae isolates, 5 Gram-positive bacterial pathogens, 5 Gram-negative nonfermenting bacterial pathogens, 9 Mycobacterium tuberculosis isolates, and 5 miscellaneous bacterial pathogens (Tables 3 and 4). The bacterial pathogens for the validation set were selected to represent various genome sizes and GC contents and to account for differences in the DNA extraction protocols. We selected ATCC strains with whole-genome sequences available from the NCBI; the NCBI sequences served as references for the validation sequencing results. Several isolates with reference sequences available from the CDC were also included in the validation set.

The genome sequences of ATCC strains, isolates characterized by the CDC, and other representative isolates were downloaded from the NCBI database (http://www.ncbi.nlm.nih.gov/genome/) to be used as a reference according to CLSI guidelines (37) (Tables 3 and 4).

Whole-genome sequencing was performed on an Illumina MiSeq sequencer (see Fig. S1 in the supplemental material). The Nextera XT library preparation procedure and 2 × 300-cycle MiSeq sequencing kits were used (Illumina Inc., San Diego, CA, USA). Illumina Nextera XT indexes were used for barcoding. Bacterial DNA was extracted by using the Wizard Genomic DNA kit (Promega, Madison, WI, USA). The bacterial DNA concentrations were measured by using Qubit fluorometric quantitation with a Qubit double-stranded RNA (dsDNA) BR assay kit (Thermo Fisher Scientific, Waltham, MA, USA). DNA purity was estimated by using a NanoDrop 2000 UV-visible (UV-Vis) spectrophotometer (NanoDrop Products, Wilmington, DE, USA). The Mastercycler nexus was used for incubation for tagmentation reaction and PCR (Eppendorf North America, Hauppauge, NY, USA). The library concentration was measured by using a Qubit HS kit. The DNA library size distribution was estimated by using a 2100 BioAnalyzer instrument and a High Sensitivity DNA analysis kit (Agilent Technologies, Santa Clara, CA, USA). Ampure beads were used for size selection. Manual normalization of libraries was performed. Genome sizes of samples pooled in one run were taken into account to ensure the equal representation of each genome per flow cell. The number of samples was kept at 16 per run (except for one run with 19 samples), resulting in a total genome load of 48 to 84 Mb/run (average, 69 Mb/run). The PhiX Control V3 sequencing control at 1% was spiked in for every sequencing run (Illumina Inc., San Diego, CA, USA). Genomes were generated with a depth of coverage (estimated for trimmed mapped reads) in the range of 15.71× to 216.4× (average, 79.72×; median, 71.55×).

The preliminary run and raw data acceptance criteria were set for (i) the percentage of bases with a ≥Q30 quality score for the run of ≥50%, with the requirement that the Q30 score for the generated genome sequences must be ≥75% for at least 80 bp of the read length; (ii) a minimum average coverage of the genome of 10×; (iii) a PhiX error rate that must be <6%; and (iv) the negative control meeting the following parameters: <10,000 reads after trimming, an N50 value of <1,000, and the highest coverage of de novo-assembled contigs of <10×. Data which did not meet these quality parameters were rejected. The preliminary quality thresholds were further adjusted based on the quality ranges observed during validation.

Paired-end reads were quality trimmed with the threshold of Q30 and then used for mapping to the reference and de novo assemblies on CLCbio Genomic Workbench 8.0.2 (Qiagen, Aarhus, Denmark). The BAM files generated after mapping to the reference genome were taken through series of software suites to generate the phylogenetic tree. A customized shell script was created to automate the subsequent steps after mapping, which included (i) calculating the genotype likelihood using SAMtools mpileup (v.1.2) (52), (ii) conversion to a VCF matrix using bcftools view (v0.1.19; http://samtools.github.io/bcftools/), (iii) single nucleotide polymorphism calling in coding and noncoding genome areas using bcftools call —c (v0.1.19), (iv) variant parsing using vcftools (v.0.1.12b) (53) to include only hqSNPs with a mean coverage of ≥30× and a minimum SNP quality (minQ) of ≥200 with indels and heterozygote calls excluded and the allele frequency defining a homozygous call at ≥93%, and (v) conversion of the SNP matrix to a FASTA alignment file for export back to CLCbio Genomic Workbench 8.0.2 for the generation of the phylogenetic tree.

Maximum likelihood phylogenetic trees were generated based on hqSNPs under the Jukes-Cantor nucleotide substitution model; bootstrapping included 100 replicates.

Genomes were annotated with prokka v1.1 (54), and species identification was performed by comparing 16S rRNA gene sequences against data in the Ribosomal Database Project (RDP) database (55).

In silico MLST was performed by using the Center for Genomic Epidemiology (CGE) online tool (56).

ABR gene detection was performed by using the CGE ResFinder online resource (57) with 99% identity and 100% query length coverage thresholds. Two sets of sequences were analyzed. For the first set, ATCC reference bacterial strains designated for use as antibiotic susceptibility controls were sequenced by the laboratory performing the validation. Negative controls were chosen among strains described as being susceptible, with no known antibiotic resistance genes, according to CLSI document M100-S25 (58). Positive controls were chosen among strains that possessed resistance determinants, according to CLSI document M100-S25. For the second set, reference sequences were acquired from the FDA-CDC Antimicrobial Resistance Isolate Bank (https://www.cdc.gov/drugresistance/resistance-bank/) for in silico testing. Thirteen isolates (7 Gram-negative and 6 Gram-positive organisms) with various resistance genes were analyzed.

Thirty-four bacterial isolates were sequenced in triplicate. The WGS pipeline, including wet- and dry-bench processes, was validated by assessing the performances of the platform, specific WGS-based assays, and the bioinformatics analysis pipeline (Fig. 1). Validated WGS assays included genome-wide SNP-based genotyping, MLST, 16S rRNA species identification, and antibiotic resistance gene detection. The following performance characteristics were assessed: accuracy, reproducibility (between-run precision), repeatability (within-run precision), analytical and diagnostic sensitivity, and analytical and diagnostic specificity.

For reproducibility assessments, all between-run replicates were generated starting from fresh cultures except for M. tuberculosis, where DNA samples were used. Between-run replicates were processed on three different days, alternating between two operators, as recommended in CLSI document MM11A (59). For repeatability, within-run replicates were prepared from one DNA extract, but independent library preparations were done, with the final samples being included in one sequencing run. The reproducibility and repeatability of the genotyping assay were evaluated with two methods: (i) evaluation of the reproducibility and repeatability of SNP calling per replicate takes into account the whole genome of one replicate as a single test, meaning that any number of single nucleotide changes in one out of three of the replicates was regarded as 33.3% disagreement for that validation sample, and (ii) the reproducibility and repeatability of SNP calling relative to the genome size were calculated by taking a discordant SNP into account as a percentage of the number of base pairs in the genome (genome size); e.g., a 10-SNP difference among the replicates with 5-Mb genomes equals 99.9998% reproducibility.

The LOD of SNP calling was estimated by modeling different mapping coverages and estimating the minimum coverage that allowed accurate SNP calling (LOD_SNP). Mapping BAM files were downsampled in order to achieve different coverage values (60×, 50×, 40×, 30×, 20×, 15×, 10×, and 5×) for each of the nine samples representative of the different species. The original sequence mapping coverage was estimated from the BAM file by using the following command: samtools depth bamfile_sorted.bam ∣ awk ‘{sum+ = $3} END {print “Average =”,sum/NR}’ The bam files were downsampled to the desired coverage by using the command samtools view —h —s F bamfile_sorted.bam >bamfile_sorted_30x.bam, where F is a fraction of the desired coverage in relation to the original coverage. Downsampled BAM files were compared to other replicates for the same sample at their original coverage.

To determine analytical specificity (interference), we generated in silico sequencing files containing a mixture of reads from two different samples, thus mimicking contamination. The effect of potentially interfering sequencing reads on mapping metrics (percentage of reads mapped/not mapped, percentage of the reference sequence covered, etc.) and SNP detection was estimated. Sample C3 (Escherichia coli ATCC 8739) was selected as a “control, not-contaminated sample,” and equal parts of reads from different species were merged with it to generate mixed fastq files. This was done by using the concatenate command, e.g., cat file1 file2 > mixed file. To ensure that the correct file was generated, the numbers of lines in both files and the mixed file were counted. The number of lines in the mixed file was the sum of the numbers of lines in the two parent files. Also, the column headings were checked to ensure that the headers of the mixed file were similar to those of the parent file. “Contaminated” and “not-contaminated” reads were mapped to the same reference genome of E. coli ATCC 8739 from the NCBI. Further SNP calling analysis was performed by using the standardized pipeline under validation. The specificity of SNP calling in the samples containing potentially interfering reads was estimated by comparing (i) the number of SNPs detected between the contaminated sequence and the reference sequence and (ii) the number of SNPs detected between the contaminated and not-contaminated sequences included in the same SNP calling analysis (vcf file after filtering).

Data from this whole-genome shotgun project have been deposited in GenBank under raw read accession numbers SRR4114366 to SRR4114399 and assembly accession numbers MTFS00000000 to MTGZ00000000 (see Table S1 in the supplemental material). All WGS data, including sequences of the replicates, are associated with BioProject accession number PRJNA341407.