We study asexual replicators, such as cancerous cells, viruses with negligible recombination, or bacteria with negligible horizontal transfer. For sexual replicators the effects of deleterious mutations are mitigated by recombination and exhibit a very different sensitivity to the mutation rate [28].

We consider whether an initial population of replicators in a novel environment leads to establishment of a successful population or goes extinct. Situations where replicators jump to a new environment often involve only a few explorers, or else abrupt environmental change can sharply decrease the size of an extant population. Also any population is most likely to go extinct when there are the fewest replicators present. For all these reasons, we focus on the dynamics of a replicator population that begins at low abundance. When there are few replicators, the interactions between them are limited, and depletion of resources is negligible. Hence for calculating the survival probability we assume that the demographic fates of replicators are independent. Consequently, we use a branching process framework [29], as is standard in the evolutionary escape literature [1], [2], [27]. We study the case of one initial replicator, but given the assumption of independence, the dynamics for initial replicators can be deduced directly from the dynamics of each of them. In particular, the probability of extinction of the entire founding population is the product of the extinction probabilities for each replicator considered alone.

To analyze the branching processes, we use generating functions which gather the information on the probabilities that a single replicator produces replicators after the next event in the system: . Standard branching process theory implies that the probability of eventual extinction is given by the smallest positive solution to (and the survival probability is ) [29]. This can be extended to multitype branching processes, with the generating map where is the probability that one replicator of strain produces a set of replicators of strain 1, replicators of strain 2, etc [29]. We assume that mutations occur upon replication.

A mutation in the genome may change fitness. In the context of adaptation to new conditions, mutations with a significant effect on replicator fitness (e.g. those that modify a reactive site on a protein for instance) can be distinguished from mutations with small effect (e.g. those that marginally alter the thermodynamic stability of a protein) [30]. Likewise, deleterious mutations can range from lethal mutations with dramatic deleterious effects on replicator fitness (e.g. a stop codon in an essential gene) to much milder effects [10], [31]. Approximating small-effect mutations as neutral mutations, we consider an idealized fitness landscape for a genome consisting of sites, where sites are lethal, i.e. the strain is non-viable if any of these sites is mutated, one site allows for adaptive mutations, and the remaining sites are neutral (figure 1). Each site represents a nucleotide, so a mutation is a nucleotide substitution, though the model framework could be modified to address allele changes across genes. We assume that the mutation rate is the same for all sites. The parameter is the probability of mutation at each site, given birth of a new replicator. We use the term “mutation rate” for consistency with the biological literature. For a real system, mutation rates can be different for different sites or for back mutations, but if a change in the mean mutation rate affects all rates proportionally (e.g. a more error-prone enzyme for replication or proof-reading), this would not affect our qualitative conclusions.

We consider a replicator of strain () which duplicates at a rate per unit time and dies at a rate per unit time. The basic reproductive number, i.e. the mean number of direct descendants of this replicator, is . We construct the generating function by considering the probability of each type of event and the resulting number of replicators of each strain. The next event is death with probability , which leads to a term in . The next event is replication with probability , which leads to a term in (for the parent replicator) (for the offspring replicator, which can be mutated as explained in figure 1: ). Thus the generating function starting from one replicator of strain is:(1)

The extinction probabilities starting from one replicator of strain are solutions of the system and .

Solving this system numerically shows the dependence of survival probability on mutation rate (black circles in figure 2). All three panels show the scenario where there is a single site where mutation is adaptive () and 10 sites where mutation is lethal (). For small enough values of (figure 2 a,b), increasing the mutation rate from a low level increases survival, as more adaptive mutants are produced. But there is a finite mutation rate at which the survival probability is maximized, because at higher mutation rates the fitness burden of lethal mutations is greater. When the initial fitness of the introduced strain is higher (figure 2c), this latter effect dominates, and any amount of mutation decreases the survival probability of the replicator lineage. Thus even when a neighboring genotype is substantially fitter, an increase in the mutation rate can be disadvantageous. To generalize this finding and derive biological insights about its determinants, we analyze the survival probabilities further for several basic scenarios.

To investigate whether mutations are beneficial at all in a given scenario, we study whether a small amount of mutation leads to more survival than no mutations, i.e. whether the initial slope of the survival probability as a function of the mutation rate is positive (orange dot-dashed lines on figure 2). We introduce a new quantity, the survival probability without mutations, which is . We can then derive approximations for the survival probability starting from one replicator of strain 1, in the limit of low mutation rate (see Appendix S1.1 in file S1). If (hereafter referred to as an “unfit” strain), the lineage dies out with certainty in the absence of mutations, and:(2)

If (hereafter referred to as a “fit” strain), the lineage can survive without mutation, and:(3)

If the coefficient of is positive in these expressions, mutations are beneficial. If the initial strain is unfit (), mutations are always beneficial, because they are necessary to have any chance to avoid extinction. If the initial strain is fit (), the presence of lethal mutants means that mutations are beneficial when the adaptive strain is much fitter, more precisely when .

This result can be generalized to the following simple rule (appendix S1.1 in file S1): calculating the survival probabilities in the absence of mutations, if the survival probability averaged over the immediate mutational neighbors is larger than the survival probability of the initial strain, then mutations are beneficial. Immediate mutational neighbors are strains one mutation away from the initial strain, which in the particular case above are 1 neighbor of survival probability and neighbors of survival probability 0. This is a sufficient condition to prove that mutations are beneficial. But it is not a necessary condition, as we will see when several mutations are needed to reach a fitter strain.

When mutations are beneficial, there is a finite mutation rate that maximizes the probability of survival for a given environmental change scenario, which we refer to as the optimal mutation rate, . To investigate how this optimum depends on the parameters of the model, we build approximations for the survival probabilities (figure 2). We define as the survival probability that accounts for lethal mutations only. A first approximation step is to neglect the back mutations from strain to strain , i.e. writing the survival probability starting from a replicator of strain as a function of the survival probability starting from a replicator of strain j, and taking as the value for the latter. The next step is , i.e. calculated using as a value for . We make further approximations based on these expressions (appendix S1 in file S1).

These approximations do not lead to a simple explicit expression for , but they do give analytical insights about the factors that influence the optimal mutation rate when the initial strain is unfit (). When the number of lethal mutants is large (), the mutation rate that maximizes survival is proportional to : as expected, the greater the frequency of lethal mutations, the lower the optimal mutation rate. Interestingly, in the limit (the mutant barely survives) does not depend anymore on , and in the limit large (the mutant is very fit) does not depend on . Thus the optimal mutation rate seems to depend only on the parameters of the strain that is closer to the threshold value governing survival, for which the fine-tuning of the mutation rate will have the largest impact on survival.

We have assumed that the risk of lethal mutations is the same for both strains. However in real systems there may be epistatic interactions such that strains have different robustness. Furthermore, from our first analysis we cannot conclude whether the results depend on the lethal mutations threatening the initial or the mutant strains. To explore this, we study a model that has two strains of differing fitness, as in Figure 1, where the initial strain is endangered by lethal mutations and the adaptive strain is endangered by lethal mutations.

Once again we determine the regime of beneficial mutations by considering the low mutation rate limit. In this limit, the survival probability of strain 1 depends on the characteristics of strain 2 only via , and thus it is independent of . Consequently, the criterion for mutations to be beneficial () depends only on , not on . If the initial strain is not endangered with too many lethal mutations, mutations increase survival.

The optimal mutation rate depends on , , and . However, by refining the initial iterative approximation in the regime (appendix S2 in file S1), we find that in the limit where both and are large, but one is much larger than the other, only the parameters of the strain threatened with more lethal mutations matter (figure 3). The same phenomenon holds qualitatively for smaller values of and . Thus to optimize the mutation rate, only the less robust strain has to be taken into account.

Often a significant increase in fitness requires more than one mutation [32], [33]. How does this affect our conclusions? Here we study a simple model where two mutations are needed to obtain a higher reproductive number , while the non-mutant and the one-mutation strains have the same reproductive number R₁, with possible lethal mutations for all strains (see figure 4a and appendix S3 in file S1). We denote the different strains by their mutational states at the two sites, from the initial strain (0,0) to the double mutant .

We consider how increasing the mutation rate affects the survival probability for a population starting from a single replicator of each genotype. Mutations are always a burden for the fittest replicator , and its survival is almost not influenced by the reproductive number of the neighboring strains as long as their survival probability is much smaller (figure 4b). The survival probability starting from a single-mutant replicator (1,0) or is very similar to the survival probability when only one mutation is required for adaptation (figure 4d; and figure S3 of appendix S3 in file S1). Starting from a replicator with no mutations , the patterns are more complex (figure 4c). If is not too large, there is a local maximum in survival probability for a mutation rate slightly larger than the of the single mutant. This arises because there is potential to reach the fitter (1,1) genotype, but the initial strain needs more mutations than the single mutant so its optimal mutation rate is higher. For very low mutation rates, however, there is a negligible chance of reaching the adaptive genotype, so if the initial strain is fit (), there is a local maximum at . This local maximum is the global optimum when is large enough, since the potential to reach the genotype is outweighed by the cost of lethal mutations, but as decreases the global optimum switches to the non-zero maximum corresponding to the strategy of adaptation. This demonstrates that our earlier criterion for mutations to be beneficial was not necessary but sufficient. If the slope of the survival probability at is positive, mutations are certainly beneficial, like before; but if the slope is negative, mutations may still be beneficial at some higher mutation rate.

Our analysis so far has assumed that deleterious mutations are all lethal, but of course fitness can decrease without going to zero [10], [15], [31]. We investigated several alternative fitness landscapes with non-lethal deleterious mutants, and found that the outcomes are very similar to our previous results (figure 5). In the limit of low mutation the results are identical, because the initial slope of the survival probability depends on the survival probabilities of mutational neighbors in the absence of mutations, so any type of deleterious mutant with leads to the same ultimate result of extinction. For larger mutation rates, deleterious rather than lethal neighbors lead to moderately higher values of the survival probability and the optimal mutation rate. The fitness of deleterious double mutants has very little influence because more than one mutation is needed to reach them. Overall, what matters most are the immediate mutational neighbors, and deleterious mutations pushing the reproductive number below one act very similarly to lethal mutations, at least at low mutation rates, because they are very likely to be evolutionary dead-ends.

Our replicator model is very general, and may need to be adapted to apply to specific systems. As an example, if we describe the dynamics of a virus within a host, a virion may have a very low probability to successfully infect a cell, but when it succeeds, the number of released virions can be large, up to at least [34]. The basic reproductive number is R = qN. When many cells are infected, fluctuations will average out and is the dominant parameter describing viral population growth. In the beginning of the infectious process, however, numbers of virions are often low [35], [36] and viral growth is fundamentally stochastic so alone may be insufficient to describe the dynamics, as emphasized by Pearson et al. in a non-evolutionary context [37].

We assume that a virion of strain successfully infects a cell with probability q_i, and that this cell has a fixed death rate and a fixed rate of production of new virions , leading to a geometric distribution of the number of new virions produced by this cell of mean . For many common viral life histories, each new virion produced by a cell may bear mutations independently of the others [38], as in the simple model above.

It appears that this description adds two more parameters to our replicator model. However, it can be shown that (appendix S5 in file S1), i.e. if we keep the reproductive numbers constant and multiply both probabilities of cell infection and by the same factor , the survival probability is also multiplied by . Thus the value of the survival probability changes, but not its dependence on the mutation rate. So when studying the dependence of the survival probability on the mutation rate, the relevant parameters are , and , as above, plus one additional parameter, , which describes how much more efficiently the mutant strain infects cells compared to the initial strain.

As in the general model, mutations are beneficial when . The survival probability of a strain in the absence of mutations is [37], so mutations are beneficial if . For fixed values of and , larger ratios lead to larger ranges of reproductive numbers where mutations are beneficial. That is, if the fitness increase is due predominantly to more efficient infection of new cells, then mutations are more likely to be beneficial.

In principle, there could be situations where a mutation that dramatically increases the number of virions produced by an infected cell () comes at the expense of the probability to infect a cell (). If is large enough, such a mutant strain has a higher average growth rate (proportional to ), but counter-intuitively a lower survival probability than the initial strain because of increased variance in the number of offspring virions produced [37].

When the initial strain needs to mutate to survive ( and ), there are two regimes for the optimal mutation rate (figure 6, appendix S5 in file S1). If , when is not too small, the survival probability is the same as for the general model, except for a factor of , leading to the same dependence on the mutation rate and the same . In the regime , an approximation for small leads to an optimal mutation rate which depends on and only. Thus the details of the life history of the virus (via the ratio ) define two regimes, but for each regime depends on the overall reproductive number only and not on and independently.