We selected study participants from the National Birth Defects Prevention Study (NBDPS), a population-based study to assess risk factors for > 30 major structural birth defects (Yoon et al. 2001). A description of NBDPS methods has been published previously (Reefhuis et al. 2015). Birth defects surveillance systems in 10 US states (Arkansas, California, Georgia, Iowa, Massachusetts, New Jersey, New York, North Carolina, Texas, and Utah) ascertained cases with eligible defects among pregnancies that ended on or after October 1, 1997, or had an estimated delivery date on or before December 31, 2011. To confirm eligibility for the NBDPS, a clinical geneticist at each NBDPS site reviewed the abstracted medical record data of cases. A clinical geneticist performed further review of clinical information to classify cases using standard case definitions across NBDPS sites (Rasmussen et al. 2003). SHFM was defined as complete or partial absence of one or more of the central rays of the hand or foot (fingers/toes and metacarpals/metatarsals 2–4, or monodactyly). Cases with SHFM were classified as having defects that were isolated (presence of one major defect) or multiple (presence of two or more unrelated, major defects in different organ systems). Cases with chromosomal abnormalities or known syndromic or single-gene disorders were not included in NBDPS.

Following the completion of a computer-assisted telephone interview within 6 weeks to 2 years after their estimated dates of delivery, mothers were mailed a kit with cytobrushes to self-collect buccal cell samples from themselves, their child (if living) and the child’s father. Mothers were excluded if they had already participated in the NBDPS with a previous pregnancy, could not complete the interview in English or Spanish, were imprisoned, or did not have legal custody of their child at the time of study enrollment. We selected participants who were the unaffected parents of NBDPS cases with non-limb birth defects and had provided buccal cell samples that were used for exome sequencing as controls for a case–control analysis.

Participants self-collected buccal cell samples using cytobrushes packaged in open paper-backed peel pouches (allowing air drying of the samples) (Gallagher et al. 2011; Jenkins et al. 2019). DNA was extracted from buccal cell samples using several protocols, as previously reported (Jenkins et al. 2019; Rasmussen et al. 2002).

Buccal cell-derived DNA samples were sequenced at the National Institutes of Health Intramural Sequencing Center (Rockville, MD; https://www.nisc.nih.gov/) in the National Human Genome Research Institute. Exomes were captured using the SeqCap EZ Human Exome + UTR kit v3.0 (Roche NimbleGen, Madison, WI), which targeted 96 megabases of the genome. DNA libraries underwent 126-base pair paired-end read sequencing on an Illumina HiSeq 2500 sequencer (Illumina, San Diego, CA). We performed image analysis, base calling, and initial mapping of sequence reads to a human reference genome to generate binary alignment map files as described in previous reports (Jenkins et al. 2019; Sok et al. 2023).

After exome sequencing and initial read mapping, the University of Washington Center for Mendelian Genomics reprocessed binary alignment map files, as reported in detail previously (Jenkins et al. 2019; Sok et al. 2023). Briefly, reads were aligned to a human reference genome (hg19hs37d5) using BWA-MEM version 0.7.10 (Li and Durbin 2010), read-pairs not mapping within ±2 SD of the average library size (~150 ± 15 base pairs for exomes) were removed, and the remaining reads underwent removal of polymerase chain reaction (PCR) duplicates, insertion/deletion (indel) realignment, base quality recalibration, variant detection, and variant quality score recalibration using software tools in the Genome Analysis Toolkit package (McKenna et al. 2010). Variant quality score recalibration involved measuring the quality of all variants using seven annotation profiles: quality by depth, mapping quality, MQRankSum, ReadPosRankSum, Fisher strand, Strand odds ratio, and inbreeding coefficient. We assessed single nucleotide variants (SNVs) and indels independently, as recommended by the Genome Analysis Toolkit, and flagged SNVs and indels below the 99.7 and 99.0 percentiles, respectively, as low quality. We also flagged variants if they had other indicators of being low quality or potential false positives, including a quality score ≤ 50, long homopolymer run > 4, quality by depth < 5, or location within a cluster of SNVs.

The University of Washington Center for Mendelian Genomics performed variant processing, as described previously (Jenkins et al. 2019). The processing pipeline included variant annotation using the ENSEMBL Variant Effect Predictor program (McLaren et al. 2016), variant filtering to retain high-quality variants with an alternate allele frequency ≤ 0.005 in population databases of genetic variants and predicted to have a medium or high impact on protein function, variant classification according to standard Mendelian inheritance models, and copy-number variant (CNV) calling using the Conifer version 0.2.2 program (Krumm et al. 2012). Variants with an autosomal recessive (homozygous or compound heterozygous), de novo, or X-linked (de novo or recessive) mode of inheritance were selected for further review and prioritized based on predicted pathogenicity. Frameshift variants, inframe indels, canonical splicing variants, and stop-gain variants were predicted to be damaging as well as missense or splice region variants that had a PHRED-scaled Combined Annotation Dependent Depletion (CADD) (Kircher et al. 2014) score ≥ 15, a SIFT (Ng and Henikoff 2003) score ≤ 0.05, a PolyPhen-2 (Adzhubei et al. 2010) score ≥ 0.85, and a Genomic Evolutionary Rate Profiling (GERP) (Davydov et al. 2010) score ≥ 3 (indicating conservation of sequence alignment across multiple species). Genes harboring these variants were prioritized based on evidence of a role in limb development or congenital limb defects in the scientific literature. We excluded variants from further consideration if they were reported as benign or likely benign in the ClinVar database (Landrum et al. 2018). CNVs with de novo inheritance and overlap with candidate genes or CNV regions previously reported for SHFM were also prioritized.

CNVs were validated using droplet digital PCR (ddPCR), performed by ACGT Inc. (Wheeling, IL). SHFM family trio DNA samples and a control human DNA sample were quantitated by the Qubit dsDNA High Sensitivity Assay (Life Technologies). Because the quantitation assay showed the SHFM family trio DNA samples had low initial concentrations (< 1 ng/μL), these samples were vacuum concentrated, resulting in an increase in DNA concentration to 1–24 ng/μL. TaqMan probes (ThermoFisher Scientific) targeting locations within the CNVs (Table S2) were used for ddPCR. The initial testing of ddPCR was performed using the QuantStudio 3D Reagent Control Kit (Applied Biology) and the TaqMan probes on the control DNA sample diluted to four concentrations: 30, 20, 10, and 5 ng/μL. After establishing that the performance of the ddPCR assays was acceptable at 5 ng/μL (Table S3), additional ddPCR assays with the TaqMan probes were performed in triplicate using the control sample at 5 ng/μL and 1 ng/μL. The assays for each probe performed satisfactorily at 1 ng/μL (Table S3), allowing ddPCR of the SHFM family trio samples to proceed in triplicate, using DNA at 1 ng/μL. The assay reaction mixture contained 7.5 μL 2X QuantStudio 3D Digital Master Mix v2 (Applied Biology), 0.73 μL 20 × TaqMan probe, 0.73 μL Reference TaqMan assay probe, 4.2 μL nuclease free water, and 1.6 μL DNA template (at various concentrations) in a total volume of 14.5 μL. The ddPCR assays were performed on a QuantStudio 3D Digital PCR System (Applied Biology), and the QuantStudio 3D Control Reagent Kit was used as an internal performance control for each run. The ddPCR assays quantitated the genomic region of interest normalized to RNaseP, an endogenous reference gene known to be present in two copies in the human genome. Results were reported as the target gene:RNaseP ratio. Ratios of 1, 1.5, and 0.5 indicate a normal copy number, a heterozygous duplication, and a heterozygous deletion, respectively, of the target gene.

Sanger sequencing was performed by ACGT Inc. using DNA samples from seven SHFM family trios to validate eight candidate variants and their pattern of inheritance. For each of the eight loci, genomic DNA (> 100 ng/reaction) was amplified by PCR (primers in Table S4), and the amplified DNA was assayed by bidirectional Sanger sequencing. Two individuals independently reviewed the sequencing results.

Prior to the analysis comparing cases with SHFM to controls, we filtered the processed exome sequence data to remove variants that were flagged as low quality, were multi-allelic, had an average read depth < 10, had a call rate < 0.99, did not meet the joint criteria of having a read depth ≥ 6 and a read depth ≤ 500 (because abnormally high read depth is indicative of alignments arising by chance alone) and a genotype quality ≥ 20 in at least 95% of samples, failed the Hardy–Weinberg equilibrium threshold at p < 10⁻⁶, or were duplicate variant records. We also removed variants that did not map to uniquely mappable regions of the genome, characterized by a 100-mers mappability score of one (Karimzadeh et al. 2018).

We excluded samples from the analysis if any of the seven evaluated metrics (mean genotype sequencing depth, number of variants called, number of singletons, inbreeding coefficient, heterozygous-to-homozygous ratio, transition-to-transversion ratio, and frequency of missing variants) were beyond ±6 SD from the mean value for all samples. We also excluded control samples that showed genetic relatedness at the second degree or closer with case children or other controls, determined by kinship estimates generated using KING (Manichaikul et al. 2010). Next, we filtered out of the analysis any variants that became monomorphic after removing these samples.

For the analysis of common variants, we included variants with an alternate allele frequency ≥ 0.05. For the analysis of rare variants, we included variants that had an alternate allele frequency < 0.05 and were predicted to be damaging (missense, frameshift, stop-gain, stop-lost, inframe indel, and canonical splice variants).

We analyzed common variants separately using logistic regression in PLINK version 1.90 (Chang et al. 2015) to identify individual variants associated with SHFM. For the analysis of rare, predicted damaging variants, we used tests for association with all variants within a gene (gene-based association testing). The subset of variants with an alternate allele in at least one case and one control (n = 49,036) was pruned, using PLINK, to retain 32,578 variants in low linkage disequilibrium with each other (r² < 0.2) for generating principal components (PCs) of unmeasured ancestry structure. All case–control analyzes assumed an additive genetic model and included sex and the first three PCs (which, together, explained 68.2% of the variance) as covariates.

The genomic inflation factor was calculated by converting p-values from the case–control association tests to chi-squared statistics with one degree of freedom and dividing the median of the resulting chi-squared statistics by the expected median of a chi-squared distribution with one degree of freedom. Because preliminary results for the genomic inflation factor indicated that population stratification was not sufficiently accounted for in the analysis of rare variants, we applied the PCAmatchR approach (Brown et al. 2021) to select 40 PC-matched controls for each case, based on the first 20 PCs, for rare variant analysis. A plot of the multivariate distance between the matched case children and controls is shown in Figure S1. As an additional measure to avoid the inflation of test statistics, we restricted rare variant analysis to genes that had at least two variants, a minimum alternate allele count of 10 in the case–control sample, and a minimum alternate allele count of 1 in cases.

We implemented gene-based testing for rare variant analysis using the optimal unified sequencing kernel association test (SKAT-O) in the SKAT version 2.2.5 package in R version 4.1.1 (Lee et al. 2012). SKAT-O aggregates score statistics for individual variants within a region and computes region-based p-values analytically. Because the number of controls was much larger than the number of case children, we ran SKAT-O using the “SKATBinary_Robust” function, which adjusts for unbalanced case–control ratios in gene-based association testing (Zhao et al. 2020). This function uses the saddle point approximation and efficient resampling approaches to calibrate the variance of score statistics for individual variants to calculate region-based p-values more accurately.

For case–control association tests, we used a Benjamini–Hochberg false discovery rate threshold of 0.05 to define statistical significance (Benjamini and Hochberg 1995). For analyzes of known SHFM susceptibility genes (Table S1) and variants within those genes, we reasoned that the tests were conducted to confirm associations with known SHFM loci. Therefore, we used an unadjusted p-value of 0.05 as the threshold to assess statistical significance at known SHFM loci.

Among 154 women with eligible pregnancies affected by SHFM, 98 (63.6%) completed the telephone interview (Figure 1). This was similar to the overall proportions of case and control mothers who completed the telephone interview in the NBDPS (67% and 65%, respectively) (Reefhuis et al. 2015). Of the 98 mothers, 53 (54.1%) submitted a buccal cell brush from at least one family member (mother, father, or child), which was a slightly lower proportion than the 65% of case mothers and 59% of control mothers who returned buccal cell brushes in the NBDPS (Reefhuis et al. 2015). Twenty-six SHFM child–parent trios had adequate amounts of DNA (≥ 200 ng for each member of a trio) for sequencing and were included in the study (case families numbered 1–26) (Figure 1). Case phenotypes and demographic characteristics are summarized in Table 1. Three case children also had another congenital anomaly (pyloric stenosis, partial pulmonary venous return, and a perimembranous ventricular septal defect) and were classified as having multiple defects in NBDPS; the remainder were classified as having an isolated defect.

We identified 119 autosomal recessive variants (one homozygous and 118 compound heterozygous), 230 de novo variants, and 97 X-linked variants (seven de novo and 90 recessive) in 305 genes in the 26 case children. Following an evaluation of the predicted pathogenicity of the variants and a literature review of the genes harboring these variants, we prioritized eight predicted damaging variants in seven genes with evidence of involvement in limb development. We visualized the eight variants in Integrative Genomics Viewer version 2.11.2 (Robinson et al. 2017) to determine if they were present on multiple reads before performing targeted sequencing to validate the variants. Each gene had a predicted damaging variant in no more than one case child.

Sanger sequencing confirmed six variants in five genes: fibroblast growth factor 13 (FGF13), interleukin enhancer binding factor 3 (ILF3), RNA binding motif protein 10 (RBM10), speckle type BTB/POZ protein (SPOP), and ubiquitin like modifier activating enzyme 2 (UBA2) (Table 2). Apart from UBA2, we did not detect variants in other known SHFM susceptibility genes. The de novo variants in SPOP and UBA2 and the variants in ILF3 were heterozygous; the X-linked variants in FGF13 and RBM10 were hemizygous.

Two heterozygous CNVs overlapping candidate SHFM regions were predicted by Conifer to have occurred de novo, each in a single case child (Figure S2). The copy-number gain in case 8 coincided with a chromosome 10q24 region, reported to be duplicated in several cases with SHFM in the literature (Everman et al. 2006; Holder-Espinasse et al. 2019). The copy-number gain in case 9 overlapped the start of the coding region and the first intron of epidermal growth factor receptor pathway substrate 15 like 1 (EPS15L1), a candidate SHFM gene located at chromosome 19p13.11 (Umair et al. 2018). ddPCR, with the use of a probe targeting the BTRC gene in the chromosome 10q24 region, determined that the BTRC:RNaseP ratio (mean ± SD) in case 8 was 1.187 ± 0.055, which was lower than 1.5 but higher than that of the parents and the control DNA sample (Table 3). In the context of mosaicism (presence of the variant in a fraction of cells in the individual), a target gene:RNaseP ratio of 1.187 could be consistent with a copy-number gain. It is uncertain if mosaicism would be dependent on cell type in the buccal cell samples, which are composed of epithelial and immune cells (Wong et al. 2022). We did not extract DNA according to cell type or determine cell type fractions in the buccal cell samples, and therefore, could not consider the distribution of cell types in the samples.

An indicator of possible mosaicism in case 8 is an allele balance (defined as the ratio of reads supporting an allele to all reads for the variant, considering only heterozygous variants) that differs between the case and the parents of the case. Therefore, allele balance in the genomic region that included the 10q24 gain and one megabase upstream and downstream of the gain was assessed in members of the case 8 family trio, using exome sequence data (Figure S3). The patterns of allele balance for heterozygous variants in the plots were similar between the case child and the parents, suggesting a similar allele balance among family trio members for variants in the 10q24 region and a lack of mosaicism in this region in the case child. Without evidence of mosaicism, a BTRC:RNaseP ratio of 1.187 would not be consistent with a copy-number gain. Thus, the predicted 10q24 gain in case 8 was not considered to be validated by ddPCR. The predicted copy-number gain overlapping part of the EPS15L1 gene in case 9 had an EPS15L1:RNaseP ratio (mean ± SD) of 0.798 ± 0.005, which was lower than 1.5 and the ratio in the control DNA sample. Therefore, this predicted variant also was not considered to be validated by ddPCR.

The initial study population for case–control analysis included 26 cases with SHFM and 1206 controls. Twenty-six cases and 1191 controls (Table S5) remained for analysis after excluding six controls that did not meet sample quality control criteria and nine controls that had a second degree or closer relationship with case children or other controls. The case children included 16 males (61.5%) and 10 females (38.5%) while the control group had 596 males (50.0%) and 595 females (50.0%) Sex was not associated with case–control status in bivariate analysis (p = 0.34; Pearson chi-squared test). We filtered the initial set of 647,419 variants for case children and controls combined to remove 211,409 (32.7%) variants that did not meet quality control criteria (Figure S4). Of the 436,010 bi-allelic SNVs and indels that remained, 18,372 (4.2%) were rare, predicted damaging variants and 17,884 (4.1%) were common variants.

Case–control analysis proceeded with the numbers of subjects, genes, and variants shown in Table 4. No genes/variants had adjusted p-values below the false discovery rate significance threshold in the analysis of rare or common variants (Figure 2a–c). The top gene/variant was the ELMO domain containing 3 (ELMOD3) gene for the analysis of rare, predicted damaging variants (adjusted p = 0.46) and a missense variant in the endoplasmic reticulum aminopeptidase 1 (ERAP1) gene for the analysis of common variants (adjusted p = 0.71) (Table 4). The ERAP1 missense variant, rs3734016, was three times more frequent in cases vs. controls (19.2% vs. 6.4%) and is predicted by both SIFT and PolyPhen-2 to be tolerated. None of the known SHFM susceptibility genes in Table S1 met the criteria for inclusion in rare variant analysis. Six common variants were detected in four known SHFM susceptibility genes and included in the analysis of common variants (Table S6). The six variants all had unadjusted p-values > 0.05; therefore, the association test results for these variants were not considered statistically significant.