We devised an example of a prevalence difference known to be epidemiologically insignificant or unimportant. The hypothesis is to test the prevalence difference of 2 interventions—A and B. To compute the P value for the test, it is assumed that intervention A reduces the prevalence of an STD by 31% in group A and intervention B reduces the prevalence of the same STD by 27% in group B. Groups A and B are equal in size, and interventions A and B are equally effective. The relative effect of intervention A to that of intervention B is 1.21 as the prevalence odds ratio. We know from practical experience that a 4% difference in the effects of interventions A and B is not clinically significant or important. Let us suppose a 20% difference would be clinically important. Although 20% may be arbitrary, is comparable to the gender gap in 2014 gonorrhea rates—120.1 cases per 10⁵ among men and 101.3 cases per 10⁵ among women.¹⁰ An intervention that closes that gap, that is, reduces the difference by 18.8%, would be epidemiologically important because closing the gap is a national goal. For our statistical simulation, the R statistical program¹¹ was used and is available as the supplementary document.

Using the outcomes described above, data can be readily simulated with different sample sizes. Figure 1 illustrates that as the sample size (N) and power increase, the P value becomes smaller, even when there is no change in the absolute difference of 4% (measure of effect). All the data points of this figure were generated with the same prevalence odds ratio of 1.21 and the absolute difference of 4%, as described in the previous section. For example, with a total sample size N of 100, P = 0.24 and appears not to be “statistically significant" using the standard threshold of P < 0.05. In contrast, a sample size N of 1800 results in P = 0.002, a value universally considered “statistically significant.” This phenomenon occurs because the P value is directly influenced by the sample size. As sample size increases, P values become smaller, crossing the 0.05 threshold to become “significant” regardless of whether the outcome is clinically significant. Without context, reporting only a P value for the group difference as evidence of its clinical significance or epidemiological importance will often result in misinterpretation. As stated above, the absolute difference in prevalence between the two groups in our example is 4%, a difference that is known to be unimportant. Thus, its effect size—the estimated difference of the prevalence rates (4%)—should be reported as well.

As shown in Figure 1, some small P values will show statistically significant differences. Instead of directly associating small P values and statistical significance with clinical significance, small P values should be interpreted as evidence supporting a rejection of the assumptions that the particular set of data are consistent with the proposed model for the data.⁹ P values in Figure 1 were modeled under a null hypothesis assuming a prevalence ratio of 1.0. P values less than 0.05 mean the data are incompatible with the model’s null hypothesis assuming a prevalence ratio of 1.0, which should be rejected.⁹ In the example provided, this is the only information the P value can provide. The incompatibility of the data and the statistical model’s null hypothesis indicated by P values less than 0.05 provides justification for a preliminary, not definitive or final, rejection of the null hypothesis.

Figure 2 shows that as the sample size N increases, the confidence interval for the effect size becomes narrower. With N =1800, confidence intervals for the estimated prevalence rates are reported as 31% to 37% and 24% to 30%, respectively. Equivalently, an estimated prevalence ratio of 1.38 can be reported with its confidence interval of 1.13 to 1.69 or its standard error of ±0.13. Neither statement about precision, however, justifies any statement about clinical significance or epidemiological importance.

In this article, we illustrated that P values measure statistical significance, but not necessarily clinical significance or epidemiological importance. The evaluation of public health interventions requires more careful epidemiological investigations including the assessment of the magnitude of attributable risks than simply reporting P values. When the P value was first proposed by Ron Fisher as an index of statistical significance, it was never meant to be used as an index of clinical significance or epidemiological importance.¹² The threshold of 0.05 was intended to serve as an initial or preliminary indicator of potential statistical significance, neither final nor confirmatory. Some disciplines, such as genomics,¹³ have used P values to support some important discoveries, but the nature of the analysis in this respect is exploratory, rather than confirmatory. Even when the P value is meant to be exploratory, the use of 0.05 as a threshold may be misleading. Observe the horizontal dotted line at a significance level of 0.05 in Figure 1. If the dotted line was drawn at a significance level of 0.1, a larger number of simulated experiments would have statistically significant P values. Regardless of statistical significance levels, however, the prevalence difference (4%) or prevalence ratio (1.21) remains clinically insignificant and epidemiologically unimportant.

As shown in the simple example presented in this article, evidence of statistical significance is not evidence of clinical or epidemiological importance. Smaller P values are not necessarily associated with larger or more important effects, and larger P values are not necessarily associated with clinical insignificance or epidemiological importance of the effect. Recall that our example had a prevalence difference of 4%, but P values in Figure 1 varied. Thus, any analysis with a large sample size or high precision may produce a small P value, whereas analyses with small sample sizes or imprecise measurements may produce large P values even though the clinical or epidemiological effect may be important.

According to the ASA, “Cherry-picking effect sizes with small P values, also known by such terms as data dredging, significance chasing, significance questing, selective inference and “P hacking,” leads to spurious significant results.”⁹ One way to prevent spurious findings is to report both effect sizes and corresponding confidence intervals so that readers can decide for themselves if the difference (or ratio) in effect is big enough to be meaningful or important on the basis of clinical or epidemiological criteria of significance. In some cases, however, effect sizes are not applicable to the result of the statistical test—e.g., a goodness of fit test only yields a P value. Therefore, there is still utility in using a P value to make decisions about a model and it should not be blindly abandoned.

Scientific writing tends to convey P values as statements about the truth of a null hypothesis, or about the probability that random chance produced the observed data. Yet, as shown in our study, whether P values are statistically rejected or not, the assumed clinical insignificance does not change. Evaluating the clinical significance of a finding is quite different from assessing the statistical significance.

Generally, P values can support exploratory or preliminary judgments about statistical significance, but they are neither certain, nor confirmatory, nor final. Without additional information, such as the effect size, a confidence interval, or standard error for context, readers of scientific reports do not have sufficient information in the P value alone to make judgments about clinical significance or epidemiological importance of the statistical finding, as clearly illustrated in our example. Recent increases in big data analytics in health science provide substantially large data sets that can produce small P values that may not be epidemiologically useful. Indeed, the influx of modern-day big data in population health for different epidemiological disciplines would require more than the P value to assess the importance of new discoveries. To make study findings easier to interpret and to enable readers to make judgments about their usefulness, we recommend that in conjunction with P values, researchers also report confidence intervals or effect sizes with standard errors.