Quantitative genetic analyses were performed using the software package SOLAR (Almasy & Blangero, 1998), a maximum likelihood-based variance decomposition approach designed for use with general pedigrees. Genetic correlation estimation assumes non-zero additive genetic variances for traits included in the bivariate analysis. To ensure this assumption was met, univariate analyses were carried out for all measurements to screen for statistically significant covariates and residual heritability (h2r) Each variable was first adjusted using an inverse Gaussian transformation to ensure multivariate normality. During univariate analysis, the “-screen” function in SOLAR was used to estimate the significance of sex, age, and the interaction of sex-by-age as covariates using likelihood ratio tests. Age was an important covariate to include due to the wide range in ages in the sample (from 3.53 to 31.44 years of age) and the potential for age to correlate with dental wear, which can impact dental dimensions (Hardin, 2019a). Age-by-sex interactions can also be important given the significant sex differences in dental morphology and therefore potential for sex differences in patterns of dental wear with age. Significant covariates (p<0.10) were included in univariate models estimating h2r

Bivariate analyses were performed in SOLAR to estimate genetic correlations between pairs of variables with significant univariate h2r estimates (p < 0.05). Covariates were included in bivariate analyses for traits for which the p-value was less than 0.10; 0.10 is the default alpha for covariate testing in the SOLAR software package. These bivariate analyses model environmental and genetic correlations (ρ_E and ρ_G, respectively) as contributors to the phenotypic correlation (ρ_P), such that: ρP=ρG(hxr2hyr2)+ρE(1−hxr21−hyr2) where hxr2 is the residual heritability of trait x and hyr2 is the residual heritability of trait y (Almasy & Blangero, 1998).

Significance testing of ρG estimates was performed in SOLAR using likelihood ratio tests, comparing models in which ρG was estimated to models in which ρG was restricted to equal 0.00 or 1.00 (or −1.00 if ρG was negative). Each ρG estimate therefore has two p-values: one describing the probability that it is equal to 0.00 and another describing the probability that it is equal to +/−1.00.

Two methods were employed to estimate phenotypic correlation matrices. In both cases, only variables included in the final genetic correlation matrix were used (i.e., those with statistically significant h2r). In the first method, carried out in R version 4.2.2 (R Core Team, 2013), data were separated by sex and then residuals from a linear regression on age were estimated for all variables and all individuals. These age-corrected data were then used to estimate Pearson correlation coefficients for each pair of variables using the cor() function, including all individuals with data for both variables. This method was used to estimate separate phenotypic correlation matrices for males and females. In the second method, phenotypic correlation estimation was performed in SOLAR during bivariate genetic correlation estimation. Phenotypic correlations estimated in SOLAR exclude variance related to kinship and statistically significant covariates (sex, age, and sex-by-age interaction), and male and female data were used in their estimation. The resulting phenotypic correlation matrix therefore accounts for the non-independence of closely related individuals in this population of rhesus macaques.

Statistical comparison of the genetic, phenotypic, male phenotypic, and female phenotypic correlation matrices was performed using random skewers analyses (Cheverud, 1996) in the R package phytools, using 5000 simulations and the “unifcorrmat” method to reduce the risk of type I error (Revell, 2012). Random skewers analysis was selected to assess the similarity of correlation matrices (following Cheverud, 1996) because it is less conservative than other methods such as common principal components analysis (Cheverud & Marroig, 2007) and can be used to assess correlation and covariance matrices.

The four estimated matrices were also compared to test matrices using random skewers analyses with the same parameters (5000 simulations and “unifcorrmat” method). Test matrices were designed to assess hypotheses for correlation matrix structure and were structured as correlation matrices with entries of 0.00 or 1.00 off the diagonal and entries of 1.00 on the diagonal. The following hypotheses were tested (Figure 1):

In addition, a functional text matrix was constructed to test the hypothesis that genetic and/or phenotypic correlations would be more positive between structures with shared primary functions. Two functions were selected, specifically to separate canine size and body size, which are hypothesized to be under sexual selection in most anthropoid primates, from those without hypothesized competitive function. Those traits hypothesized to be under sexual selection, referred to hereafter as competition-related traits, were dimensions of the canine-premolar honing complex and long bones (Galbany, Tung, Altmann, & Alberts, 2015; Leutenegger & Kelly, 1977; though see Blomquist & Turnquist, 2011; Kimock, Brent, Dubuc, & Higham, 2022; Kimock, Dubuc, Brent, & Higham, 2019). All remaining traits (mandibular dimensions and dental dimensions outside the canine-premolar honing complex) were deemed mastication-related traits. So the following hypothesis was tested:

Patterns of correlation were visualized and described using complete-linkage hierarchical clustering of correlation matrices with the R package corrplot (Wei et al., 2021). Eigenvalues and eigenvectors of all matrices were estimated (following Rohlf, 2017) using the eigen function in R.

Measurement repeatabilities ranged from 69.2% (P₄MD) to 99.9% (mandibular ramus depth) (Table 1), and repeatabilities for all but four measurements were greater than 90%. All measurements were retained for additional analyses to ensure that all regions of the tooth row were represented, but findings for low-repeatability measurements (P³MD, P⁴MD, P₄MD, and M¹BL) should be interpreted cautiously. The measurement data from this study are provided (Supplementary Table 4).

Univariate quantitative genetic parameter estimation in SOLAR produced residual heritability estimates (h2r) for all 40 traits (Table 1, Supplementary Table 1). Of the 40 traits, 26 had estimates significantly different from zero (p < 0.05). Estimates of h2r ranged from 0.00 (C¹ height, C₁ height) to 1.00 (P₄ height), with a mean value of 0.479. Sex was a significant (p < 0.1) covariate for all traits. Age was a significant covariate for 26 traits in total, and for 17 of the 26 traits with significant h2r. Sex-by-age was a significant covariate for 11 traits in total, and for 6 of the 26 traits with significant h2r. Variance accounted for by covariates ranged from 0.073 (P₄ breadth) to 0.697 (geometric mean of post-cranial measurements). Kurtosis values for these analyses ranged from 0.005 to 3.875.

By tissue type, h2r ranges and means were similar; for non-dental traits the mean h2r value was 0.455 (range: 0.111 – 0.859) and for dental traits the mean h2r value was 0.487 (range: 0.00-1.00). By function, masticatory traits had higher average h2r values; for competition-related traits the mean h2r value was 0.339 (range: 0.00-0.790) and for mastication-related traits the mean h2r value was 0.555 (range: 0.249-1.00). By anatomical region, post-canine traits had the greatest h2r values; for canine-premolar honing complex traits the mean h2r value was 0.322 (range 0.00-0.790), for post-cranial traits the mean h2r value was 0.369 (range: 0.111-0.636), for mandibular traits the mean h2r value was 0.541 (range: 0.274-0.859), and for post-canine traits the mean h2r value was 0.558 (range: 0.249-1.00). The fourteen traits with non-significant h2r were excluded from all following analyses.

Bivariate quantitative genetic parameter estimation in SOLAR produced 325 genetic correlation estimates (ρG) for the 26 traits with significant h2r estimates. Estimates of ρG ranged from −0.404 to 1.00, with a mean of 0.418. Of the 325 ρG estimates, 49 values were significantly different from 0.00 and +/− 1.00, 68 values were significantly different from 0.00 but not from 1.00, and 183 values were significantly different from +/− 1.00 but not from 0.00 (Supplementary Table 2). 25 values were not significantly different from 0.00 or +/− 1.00 and were therefore inconclusive. No negative ρG estimates were significantly different from zero. Eight ρG estimates were equal to 1.00.

Pooled phenotypic correlations, estimated in SOLAR, ranged from −0.139 to 0.808 with a mean of 0.288. Male-specific phenotypic correlations, estimated in R, ranged from −0.138 to 0.858 with a mean of 0.333. Female-specific phenotypic correlations, also estimated in R, ranged from −0.219 to 0.797 with a mean of 0.264.

Random skewers analyses of the estimated matrices found significant similarity among all four matrices (p<0.001). The greatest r-value from these comparisons was between the male-specific phenotypic correlation matrix and the phenotypic correlation matrix estimated in SOLAR (r = 0.956). The genetic correlation matrix had the greatest r-value with the male-specific phenotypic correlation matrix (r = 0.846) and the smallest r-value with the female-specific phenotypic correlation matrix (r = 0.813). Across all comparisons, r-values were greater between phenotypic correlation matrices than between genetic and phenotypic correlation matrices (Table 2). The estimated matrices were all more similar to each other, based on random skewers analyses, than they were to any of the test matrices.

Despite these similarities, not all of the estimated matrices were equally similar to the test matrices. Random skewers analyses between estimated matrices and test matrices yielded r-values ranging from 0.374 to 0.809 (Table 3). All four estimated matrices were significantly (p<0.05) similar to the test matrices for integration (Figure 1B), tissue type (Figure 1C), and anatomical region (Figure 1D). Each estimated matrix was most similar to the tissue type test matrix, although the male-specific phenotypic correlation was nearly equally similar to the tissue type and anatomical region test matrices (r = 0.809 and 0.807, respectively). All phenotypic correlation matrices were significantly similar to the function test matrix (Figure 1E), but the genetic correlations matrix was not. Only the female-specific phenotypic correlation matrix was significantly similar to the independence test matrix (Figure 1A; r = 0.581). Eigenvalues and eigenvectors for all nine matrices are also provided (Supplementary Table 3).

In all estimated correlation matrices except the phenotypic correlation matrix estimated in SOLAR, post-cranial and mandibular traits clustered together and separately from any dental traits (Figure 2). The order of dental trait clustering was slightly different for each of these matrices. Tooth mesiodistal lengths clustered separately from buccolingual breadths in the female- (Figure 2C) and male-specific (Figure 2D) phenotypic correlation matrices, with C₁ length somewhat separate. There was no clear organization to dental trait clustering in the genetic correlation matrix (Figure 2A). In the phenotypic correlation matrix estimated in SOLAR (Figure 2B), all molar mesio-distal lengths except M³ length clustered with all post-cranial and mandibular traits, as did P₃ height.

Skeletal traits are generally expected to demonstrate greater environmental variance and therefore lower heritability than dental traits due to lifelong remodeling of bone (Raggatt & Partridge, 2010), yet heritabilities were not simply greater in dental traits than in skeletal traits in this study. Post-canine dental h2r estimates were greatest on average (mean h2r=0.56), followed closely by mandibular skeletal h2r estimates (mean h2r=0.54). Post-cranial skeletal h2r estimates were lower on average (mean h2r=0.37) as were canine-premolar honing complex dental h2r estimates (mean h2r=0.32). The low heritability of canine-premolar honing complex traits is partially explained by the inclusion of tooth crown heights, which tend to be more severely impacted by tooth wear than tooth breadths or lengths and had smaller sample sizes with inconclusive heritability estimates (h2r=0). Estimates of h2r were higher for mandibular traits in this study than in a previous study of mandibular distances in baboons (Willmore, Roseman, Rogers, Cheverud, & Richtsmeier, 2009), but were similar to previously estimated heritabilities of craniofacial traits in baboons (Roseman et al., 2010). The greater h2r of mandibular traits than post-cranial traits indicates that variation in mandibular dimensions is more strongly genetically influenced than is variation in non-mandibular skeletal measurements in this population. This trend may be unique to this population, but it could reflect a broader tendency in Macaca, papionins, or even cercopithecoids.

Stress, disease, and nutrition can all impact skeletal growth and remodeling to increase environmental variance and thereby reduce heritability in skeletal traits. The Cayo Santiago rhesus macaques are somewhat shielded from these sources of variance because of dietary provisioning, yet studies of linear enamel hypoplasia do indicate rank-related impacts on individuals’ nutrition and stress in this population (Benderlioglu & Guatelli-Steinberg, 2019). The familial aspects of rank may bias heritability estimates (Bourret & Garant, 2017), though the sample includes multiple social groups and matrilines that may help to offset that potential bias. Many individuals included in the sample for this study were also born prior to 1956 when consistent provisioning began on Cayo Santiago (Guatelli-Steinberg & Benderlioglu, 2006). It is therefore difficult to estimate the relative impacts of rank and provisioning on the heritability of dental and skeletal traits in this population. Studies of the same traits in closely related provisioned and non-provisioned pedigreed populations would be necessary to test the hypothesis that features of the masticatory apparatus are highly heritable even in a population with greater individual variation in dietary quality. Comparing heritabilities across populations is also fraught due to the potentially weighty impacts of unaccounted for genetic and non-genetic differences (Falconer & MacKay, 1996).

Genetic correlations were positive or near zero across most measurements, and genetic correlations tended to be greater within dental, mandibular, and post-cranial traits than across these trait categories (Figure 2). Previous studies have estimated genetic correlations between mandibular and craniofacial distances in baboons and humans (Joganic et al., 2018, 2012; Sherwood et al., 2008; Willmore et al., 2009), but have not estimated genetic correlations across dental, mandibular, and post-cranial traits. One study of a human population estimated genetic correlations across tooth areas and arch measurements and found strong positive genetic correlations between post-canine tooth areas and arch length (Hardin et al., 2022). In the present study, mandibular corpus length was highly genetically correlated with post-cranial dimensions and M² breadth, but the relationships between post-canine tooth sizes and mandibular corpus length were not as pronounced in this macaque population as in the aforementioned human study.

A study of genetic correlations between body size and molar size in baboons estimated statistically significant positive ρG estimates between body size and molar buccolingual breadths, but not between body size and molar mesiodistal lengths (Hlusko et al., 2006). That same pattern was not observed in this population of rhesus macaques. In fact, mesiodistal lengths clustered more closely with post-cranial dimensions than did buccolingual breadths based on hierarchical clustering. The only dental traits with statistically significant positive ρG estimates with post-cranial measurements were P₃ height, P³ length, and M₃ length. The mean ρG between molar mesiodistal lengths and the post-cranial geometric mean was 0.309 (range: −0.108 – 0.509), slightly greater than the mean ρG of 0.305 (range: 0.109 – 0.421) for molar buccolingual breadths and the post-cranial geometric mean. A few important differences between this study and that of Hlusko et al. (2006) may explain these differences in the findings. Hlusko et al. (2006) had a larger sample of 630 baboons, used body mass and trunk length collected from living individuals, and screened body mass and trunk length as covariates during quantitative genetic analyses. This sample was smaller at 362 macaques and femoral and humeral joint dimensions served as measures of body size in the present study. Body size was also not included as a covariate in this study because this inclusion could potentially bias heritability and genetic correlation estimates by removing informative covariation. One challenge of comparisons across populations is that so few populations have the large sample sizes and pedigrees necessary for quantitative genetic parameter estimation, and those that do may differ in the availability of skeletal or morphological data. Hlusko et al. (2006) studied a population of living baboons and did not have access to skeletal data that are available from the Cayo Santiago macaques, but this study of the Cayo Santiago macaques likewise did not have access to body mass and trunk length data that can be gathered from a population of living individuals.

Genetic correlations were not especially high between post-cranial measurements and post-canine tooth dimensions. Dental traits are frequently used to estimate overall body size in fossil assemblages (Gingerich et al., 1982; Gould, 1975; Hopkins, 2018; Millien & Bovy, 2010), yet previous research has shown that covariation between tooth and body size is not especially strong within mammal populations (Copes & Schwartz, 2010). The weak genetic correlations estimated here are consistent with the hypothesis that tooth-body size relationships are maintained through selection on metabolism and body scaling, rather than through pleiotropy. The dental traits most closely genetically correlated with post-cranial size (P³ length, P₃ height, and M₃ length) are among those that complete eruption in the fifth year or later, though fourth premolars and canines are also late-erupting teeth in this population (Wang, Turnquist, & Kessler, 2016). Additional study of the quantitative genetics of developmental parameters would contribute to the understanding of relationships among these late-developing traits.

Random skewers analyses were used to compare the estimated genetic and phenotypic correlation matrices to five test matrices representing the following five hypotheses: 1) complete independence across all traits, 2) complete integration across all traits, 3) complete integration within tissue types (tooth and bone), 4) complete integration within anatomical regions (canine-premolar honing complex, post-canine, post-cranial, and mandibular), and 5) complete integration within functional groups (competition and mastication). The genetic correlation matrix and all three phenotypic correlation matrices were most similar to the tissue type test matrix (Hypothesis 3). The genetic correlation matrix was least similar to the test matrix reflecting complete independence (Hypothesis 1). These findings indicate that traits are genetically and phenotypically correlated in this population, especially within tissue types, but that genetic correlations cannot be assumed to maintain scaling relationships between tooth and bone. The weakly positive genetic and phenotypic correlations estimated here between dental and skeletal traits may be sufficient to maintain efficient mastication, and if they are not, that indicates that there may be a limit to the degree of genetic correlation that is possible between different tissue types, due either to the existing genetic and developmental pathways upon which they are founded or to potential evolutionary costs of their integration (e.g., Evans et al., 2017; Schluter, 1996; Wagner & Altenberg, 1996).

The genetic correlation matrix was very similar to all three phenotypic correlation matrices, providing support for the conjecture that phenotypic correlation matrices may serve as useful approximations of the genetic correlation matrix when it cannot be estimated (Cheverud, 1988). The similarities indicate that environmental effects are contributing to covariation in the same direction as genetic effects for most traits. There were, however, several important differences between the genetic and phenotypic correlation matrices, including instances of negative environmental correlations in traits with significantly positive genetic correlations. These negative environmental correlations were typically reducing highly positive genetic correlations, resulting in near-zero phenotypic correlations (e.g., C₁ length and M₁ length, M₃ length and M₃ breadth, geometric mean and corpus depth). Taking the correlation between the geometric mean of post-cranial measurements and mandibular corpus depth as an example, the results demonstrate pleiotropy between the two traits, indicating that some alleles that would increase post-cranial size would also increase mandibular corpus depth. The negative environmental correlation indicates that some combination of non-genetic effects that would increase post-cranial size would also decrease mandibular corpus depth. When negative environmental correlations occur (e.g., an experimental manipulation on one trait that could reduce its correlations with other traits), the phenotypic correlation matrix will be dissimilar to the genetic correlation matrix and substitution is not recommended.

The genetic correlation matrix was most similar to the male-specific phenotypic correlation matrix based on random skewers analyses. These two matrices also tended to have greater correlation estimates than the other estimated matrices. High phenotypic correlations do not necessarily indicate greater genetic correlations in males than in females, because greater environmental correlations could also increase phenotypic correlations. The differences in phenotypic correlations between males and females is likely due to differences in the overall degree of integration between the sexes, and not due to differences in the sample sizes since a similar number of males and females were included. Unfortunately, the sample size for this population is not yet sufficient to estimate sex-specific quantitative genetic parameters. The female phenotypic correlation matrix was very similar to the male phenotypic correlation matrix, but it was characterized by a greater degree of independence between dental mesio-distal lengths and bucco-lingual breadths.

Differences in male and female dental and skeletal development could contribute to sex differences in environmental and genetic correlations. Post-cranial epiphyses fuse earlier in females than in males (Cheverud, 1981; Turnquist & Kessler, 1989) and adult body mass is achieved at 5.5 to 6.5 years of age in females and at 6.77 to 7.78 years of age in males (Turcotte et al., 2022). Males reach a greater adult body size than females through both prolongation of growth and a faster growth rate for body mass (Turcotte et al., 2022). Male rhesus macaques experience more rapid growth of the craniofacial skeleton from 2 to 8 years of age, while females experience a longer period of growth from 8 to 15 years of age (Wang, Dechow, & Hens, 2007). Timing of dental eruption is generally similar in males and females although canines erupt later in males than females (Cheverud, 1981). These developmental differences likely contribute to sex-specific patterns of environmental correlations. Environmental conditions during a period of especially rapid growth (i.e., a sensitive developmental window; McPherson, 2022; Fawcett & Frankenhuis, 2015) in males could increase environmental correlations among traits actively growing during that spurt. Those same traits may be subject to a longer, more variable period of growth in females and would therefore exhibit weak or even negative environmental correlations. For example, an environmental stressor like a hurricane (e.g., Testard et al., 2024) during a sensitive developmental window for a male individual could reduce the growth of the mandible, long bones, and teeth and result in a positive phenotypic correlation among these traits even in the absence of a positive genetic correlation. A female experiencing the same stressor could exhibit reduced growth of the long bones alone, if growth of the teeth and mandible had ceased, and would therefore have lower phenotypic correlations across these traits than the male. Growth and development is also closely associated with evolutionary tradeoffs for males and females, leading to the evolution of sex-specific patterns of growth and development (Altmann & Alberts, 2005). These differences in growth could explain the overall greater correlations in the male phenotypic correlation matrix relative to the female correlation matrix.

The similarity between male and female phenotypic correlation matrices is potentially surprising given the substantial morphological differences between male and female rhesus macaques. It is possible that the phenotypic correlation matrices are more similar to each other than are the underlying genetic correlation matrices. Sex differences in patterns of genetic correlation would indicate reduced between-sex genetic correlations. Most traits are assumed to have high between-sex genetic correlations because male and female genomes are almost identical (Poissant, Wilson, & Coltman, 2010). When between-sex genetic correlations are high, the same alleles contribute to the same traits in the same direction in males and females. In cases of sexually antagonistic selection, in which an allele is beneficial in one sex but detrimental in the other sex, sex-specific patterns of gene expression are expected to evolve. These patterns would reduce between-sex genetic correlations and allow traits to evolve somewhat independently in males and females (Lande, 1980). This has been observed in primary reproductive structures and fitness components, and between-sex genetic correlations tend to decrease as sexual dimorphism increases (Poissant et al., 2010). This is a potentially informative area of future study in this population as the skeletal collection and its pedigree grow.

Investigations of scaling relationships between body and tooth size have demonstrated that individual variation within species and populations does not adhere closely to scaling relationships apparent at higher taxonomic levels (Copes & Schwartz, 2010). The results of this study illustrate how genetic correlations in a single species relate to those scaling relationships, first within dental traits and then between dental and post-cranial traits. The tendency for strong genetic correlations across dental, especially post-canine, measurements in this species is consistent with the identification of consistent molar proportion scaling relationships at many taxonomic levels (Bermúdez de Castro, Modesto-Mata, Martín-Francés, et al., 2021; Billet & Bardin, 2021; Halliday & Goswami, 2013). If these relationships are maintained genetically via pleiotropy, they are observable phenotypically at all taxonomic levels because very few individuals develop without the influence of this highly conserved genetic architecture.

On the other hand, there are weak genetic correlations between most dental measurements and most skeletal measurements despite the important functional and developmental relationships between the skeleton and dentition. Scaling relationships tend to be apparent at higher taxonomic levels in primates despite these low genetic correlations (Copes and Schwartz, 2010). Differences in the genetic expression of dental and non-dental tissues may preclude the establishment of close genetic correlations. Necessary functional relationships between dental and non-dental tissues would therefore be maintained primarily through selection rather than pleiotropy. This would be consistent with the tendency for body and tooth size to be closely correlated at higher taxonomic levels when individual variation is averaged out of the analyses. The individual variation that is possible because of lower genetic correlations between dental and skeletal tissues would also allow natural selection to act rapidly on tooth-body size relationships.

The weak genetic correlations between dental and skeletal traits may initially seem discouraging for those interested in estimating body size from dental remains, but phenotypic correlations between skeletal dimensions and molar lengths are quite high and the phenotype is typically the goal of body size estimation. These results do, however, demonstrate the value of distinguishing between interspecific and intrapopulation relationships when estimating body size from dental dimensions, especially when examining small samples as is often the case for fossil primates.