Recent escalation of national bed net distributions and universal coverage campaigns stimulated concern and investigation into the integrity and durability of long lasting insecticidal nets (LLINs) under ordinary use conditions. Mass distribution campaigns are recommended to deliver LLINs to households, rapidly increasing population coverage and improving equity of bed net ownership [1]. Attrition occurs following these campaigns, and LLINs degrade due to ordinary use and washing, compromising the physical integrity and chemical composition of the bed net. As a result, malaria control programs are challenged with devising LLIN replacement strategies that include considerations for net degradation and attrition in order to maintain high levels of protective coverage in the community.

In 2005, the World Health Organization Pesticide Evaluation Scheme (WHOPES) developed “Guidelines for laboratory and field testing of LLINS” [2]. This document focused on measuring the bioefficacy of LLINs. Methodologies to test the bioefficacy of LLINs have been thoroughly developed and standardized, including laboratory investigations using cone bioassay and tunnel tests [3] [4]. Moreover, following these guidelines, a variety of experimental hut studies have been performed to investigate the penetration through, feeding success, and mortality of mosquitoes among treated and untreated nets with and without holes [5–18].

In 2011, WHOPES published a second document “Guidelines for monitoring the durability of LLINs under operational conditions” [19]. In addition to the previously described measures of insecticidal activity (bioefficacy), these guidelines suggest two additional outcome measures to assess the useful life of a LLIN: a general measure of attrition and the physical condition or fabric integrity. However, until the recent mass distribution of LLINs, concern about the durability of bed nets in actual field conditions and investigation of the integrity and physical durability of the fabric have been lagging.

The integrity of different fabrics in laboratory settings have been compared using standard burst and tear strength tests identifying differences between weight, textures, and weave patterns [20]. Techniques to measure fabric integrity in the field; however, can be laborious and challenging. Precise laboratory measures of irregularly shaped holes are often impractical in general field conditions. Bed nets are often hanging in dark rooms where access may be a concern for privacy and where visibility is poor, making it difficult to detect holes. Some owners choose to take down the nets before allowing them to be evaluated which is not only time consuming, but also may risk the household neglecting to re-hang the nets after the evaluation is completed. Even with good visibility, it is possible to lose track of which holes were already counted on bed nets with a large quantity of holes. Moreover, there is often confusion on how to quantify and categorize different sized holes.

While challenges of accurately determining the counts, sizes, and locations of irregularly shaped holes on bed nets exist, there is an even greater need for a composite score or standard measure that quantifies the overall general physical condition of the net to address the question: “How do you determine when a net is no longer able to protect an individual sleeping under it from a malarious mosquito?” Until recently, there has been no consistent methodology for evaluating holes in nets that could serve as a standard by which net protectiveness could be measured. For example, is a net still protective with 2 small holes and one large hole, and how does that compare to a net with 4 medium sized holes? Prior to the development and routine use of the WHOPES recommendations, this question was addressed in many different ways with each study using unique approaches to measuring durability including different measurement tools, hole size categories, weights, algorithms, and thresholds. A few studies presented results of hole counts without a composite index of the physical durability [21–23]; however, most researchers developed a variety of simple algorithms to generalize the physical condition of the entire net. These include a weighted score [24–29], predetermined cutoffs such as ‘at least one hole’ [30–34] or ‘at least a certain number of holes of a specific size’ [35–42], size of largest hole [43], and sum of the perimeters of the holes[44]. Qualitative measures have also been suggested, for example Mutuku et al. reports that respondents perceived condition of the net may be more important than measuring the actual physical condition of the net [27]. These inconsistencies in assessing physical durability across studies not only make it nearly impossible to compare different net brands, years, and locations, but also make it difficult to conclude what constitutes a failed net.

Protection of an individual by a decaying net may decrease gradually, making it challenging to know exactly when a net is no longer functional and needs replacement. However, malaria control programs need to determine the optimal time to replace an entire population of nets. Many malaria control programs are replacing nets at three year intervals on the assumption that most nets will not be functional after that time. However, if the estimate of what is a “failed” net are too stringent, it means programs are replacing nets sooner than needed. If the estimate is too forgiving, then programs may be leaving people with nets that provide inadequate protection for an extended period of time. By incorporating information on the physical condition of LLINs in field situations and devising a method to better estimate a threshold of LLIN damage, programs may be better able to determine on average when a net fails and can design appropriate replacement strategies.

The 2011 WHOPES guidelines provide a standardized protocol to assess physical condition of LLINS in field situations. Included in this guidance is a composite measure of the nets’ physical condition called the proportional hole index (PHI) which is intended to provide an estimate of relative net damage for comparison among nets. This measure categorizes holes into four sizes. The number of holes of each size are counted and recorded during field evaluations. The PHI is calculated by multiplying the number of holes in each category by the corresponding weight (based on the estimated midpoint area of each hole category) and summing this across categories. Several studies have already reported bed net durability in field conditions using this methodology [26, 45–49] [50].

The PHI provides a much needed standardized approach to assessing the physical condition of a bed net in operational conditions. However, in order to develop a simple universal algorithm several generalizations are made on the size, shape, and location of holes. First, holes of all sizes are generalized into four categories based on an easy to measure finger/fist/head methodology. Holes > 25 cm in diameter are lumped together into one category, and holes < 0.5 cm are not counted. Second, the PHI assumes all holes to be circular when estimating the area of the holes. Third, the distribution of the holes of different sizes throughout the net is considered to be uniform. In other words, the PHI does not take into account the locations of the holes on the net. It is important to understand how these generalizations may influence the overall index and interpretation of the relative conditions of the nets. Moreover, the question remains, “at what PHI value is a net considered no longer protective?” Following the creation of the PHI, several studies have selected a cutoff (varying from 88 [27], 300 [26, 45, 46], to 768 [49]) as the hole index level at which a net is deemed no longer protective. Recent recommendations of the Vector Control Technical Expert Group at the Malaria Policy Advisory Committee Meeting (September 2013) proposed the following categories and PHI cut-offs to classify a LLIN: 0–64 good, 65–642 damaged, and 643+ too torn [51]; however, additional research is warranted.

This study investigates how the size, shape, and location of holes influence a composite hole index of overall net durability for two different brands of bed nets collected three years following a mass distribution campaign in Nampula Province, Mozambique. The present study uses data from a prospective evaluation of LLINs tagged during the mass LLIN distribution campaign in October 2008. The evaluation consisted of three annual cross-sectional surveys in either October or November of 2009, 2010, and 2011 and assessed the physical integrity of the LLINs following the WHOPES guidance as well as a more thorough laboratory measure[50]. In this paper we will explore the impact of these three generalizations (hole size, shape, and location) made using the current WHOPES PHI based on the availability of more detailed lab measurements.

The physical durability and usable lifetime of a bed net is of programmatic interest in order to determine the most cost effective timing and distribution of replacement nets. According to WHOPES an LLIN is a product that retains biological efficacy for at least 3 years, defined as 80% of nets with either ≥ 95% knock-down or ≥ 80% mortality [52]. This definition is entirely based on the measurement of insecticidal activity in the nets after general washing and use but does not contain criteria for some measure of fabric integrity including burst strength. Evidence from recent investigations following LLIN mass distribution campaigns suggests that the actual lifetime of the nets in the field is shorter than expected [48].

The 2011 WHOPES guidelines provide a much needed standardized approach to assessing the practical lifetime of a bed net in operational conditions. They not only help malaria control programs devise timely replacement strategies, but also provide programs with a quantitative approach to compare different fabrics and determine the most appropriate and cost effective bed nets for their specific situation. However, the proportional hole index makes several generalizations related to size, shape, and location of holes that limit the utility for both of these objectives. Public health practitioners may want to consider additional methods to help optimize the use of the PHI for specific program goals when applying the WHOPES assessment in the field.

When the primary program goal is only to determine whether or not a group of nets are in good condition or need replacement, it may not be necessary to count all of the holes in every net. Instead one could just count holes up until a specific threshold is met. Results of this study found that roughly 75–80% of the nets exceeding a PHI cutoff could be identified just by counting holes larger than a fist which are easy to identify in field conditions. Depending on the designated threshold for determining a net no longer protective, a simple guide or decision tree may be created to facilitate bed net assessments and reduce the number of holes counted per net as well as the number of nets that need all of their holes counted. The simplicity of this method may make it suitable to be incorporated in large household surveys such as a malaria indicator survey. In addition, this type of simple hole counting method could be used for study designs such as lot quality assurance sampling as part of a program’s monitoring strategy for bed net replacement.

Even if a simplified method such as a decision tree or a more thorough method using a PHI cutoff is promoted, the range of hole areas within the simple WHOPES categories may lead to some inconsistencies. For example, a cutoff that is based on 5 or more large holes (or a PHI of 800) may have unintended consequences. It is possible that a net with 4 large 24 cm holes (~ 1808.6 square cm, PHI = 784) would be classified as still functional, whereas a different net with 5 large 10 cm holes (392.5 square cm, PHI = 980) would be classified as needing replacement even though the total damaged area is more than four times larger in the former net. Since the actual size of the hole is not recorded, just the number of holes in each category, this inconsistency would be largely undetected.

Moreover, an important question of interest “what cut-off(s) should be used to determine when a bed net or LLIN population has reached the ‘end of useful life’?” remains outstanding. The recent recommendations of the Vector Control Technical Expert Group at the Malaria Policy Advisory Committee Meeting proposed PHI cut-offs based on existing evidence to classify a LLIN into the following categories: 0–64 good, 65–642 damaged, and 643+ too torn [51]. However, gaps in knowledge still exist in this area and many unresolved issues about what hole sizes, shapes, and locations are most important for mosquito penetration and malaria transmission have yet to be answered.

When the primary program goal has a more research focused objective to develop a system for evaluating existing bed nets to determine which model may most suitable in a particular area a more thorough investigation of the net condition may be necessary. This objective often requires a more rigorous research approach performed on a limited number of bed nets to understand the formation of holes; their size, shape, location, and cause of damage as well as estimate the total damaged areas of bed nets over their lifetime. In these cases improvements on the PHI may be warranted to get a more accurate picture of bed net damage, as well as to properly assess a new model, weave, or user setting and location and especially when new brands are introduced into the market.

Results from this study show hole sizes were not symmetrically distributed within hole categories, but decreased in number as the size increased. As a result, the PHI overestimated the amount of holed surface area for small, medium and large holes. In contrast, extra-large holes were found to be underestimated compared to circular estimates. When considering the PHI composite measure, it was found to overestimate area for all years when compared to an elliptical based measurement. In addition it overestimated the area in polyester nets in all three years when compared to circular estimates using actual measured axes. However, the PHI underestimated the actual measured area in polyethylene nets for 2010 and 2011 compared to circular estimates, indicating that underweighting by using the midpoint from the extra-large hole categories has a greater impact (though fewer holes) than the overweighting from the small, medium, and large categories. This finding was more prominent in polyethylene than polyester nets as they tended to have more extra-large holes.

Extra-small holes (less than ½ cm) are excluded in the WHOPES guidelines. They are likely too small for easy passage of mosquitos and therefore when considering efficacy of an LLIN, the net would still be considered an effective barrier for its user even with an extra-small hole. However, when explicitly considering the physical durability of LLINs in the context of lifespan and replacement, extra-small holes suggest overall wear on the LLIN, can unravel leading to larger holes, and may serve as an indicator of a reduced lifespan of the net compared to one without extra-small holes. Extra-small holes are often very challenging to observe and measure during field studies [53], as such, it may be beneficial to use a proxy to estimate their numbers such as the number of other sized holes on a net. A plot of the relative proportion of the number of extra small holes on a net compared to the number of all other sized holes on a net revealed a difference between brands. While polyester nets show an increasing number of extra-small holes with an increasing number of all other sized holes, polyethylene nets have fewer extra small holes relative to increasing numbers of larger sized holes. There are many potential explanations for this finding. One such explanation would be that due to weave patterns, a small hole in polyethylene may more rapidly expand to become a larger hole, so the likelihood of observing a small hole on polyethylene nets is reduced.

The formula for the PHI assumes holes are circular; and in the field a hole is categorized based on whether one can insert a finger, fist, or head through the hole. These measures overlook the possibility of irregularly shaped holes. This study found that most holes were 2 to 10 times longer in one direction (more long and narrow than circular), and as result the area assuming the hole is circular tends to overestimate the actual holed area by roughly 2 to 10 times. It is possible that the formula of an ellipse better approximates the true holed area; however, this too does not account for irregularly shaped holes and is laborious to obtain.

The largest proportion of holes was found to be located at the bottom 1/4 of the net. Certain sections of a net may be more prone to tears based on use or other factors; also certain areas of the net may be more likely for mosquitos to penetrate than others. By understanding what parts of nets are most vulnerable, measures can be taken to strengthen nets at these locations. Future studies may want to take into account not only the size of the hole, but also the location when calculating a composite index scores by weighting or adjusting the score based on the location of the hole (e.g. weight holes on the top of a net more than those on the bottom of the net if future studies show mosquitos are more likely to enter holes in the top of the net). Also note that the PHI ignores repairs in nets such as stitches, knots, patches.

There are some limitations of the current study that should be noted. This study was initiated prior to the initial release of the WHOPES guidance and as a result several of the logistical methods did not follow the finger/fist/head recommendations exactly. However, this analysis consisted of the more precise lab measurements, and should not be affected by this deviation in the design. Lab recorders were requested to measure the longest section of a hole (major axis), and for the 3^rd year also the perpendicular to the major axis (minor axis). It is possible that for irregularly shaped holes some measurement error was introduced. In addition, the shape of holes varied greatly, and approximating the holes as a circle or ellipse has potential for error. This study took place prospectively over the course of three years, during which 16.6% of the households did not retain all of the bed nets received during the campaign. A survivorship bias may have been introduced into this data if the condition of the nets lost due to attrition differed from that of nets measured in this evaluation. In addition, the extra-small hole analysis only took place on nets during the 3^rd year of data collection, and results may not be representative for younger nets.

This paper demonstrates that measuring the physical integrity of bed nets is multifactorial and requires a thorough understanding of the size, shape and location of holes. More research is needed to potentially refine the PHI measurement tool as well as improve the manufacture and development of more durable bed nets. Moreover, physical integrity is not the only determinant of overall protectiveness of a bed net and it is important to include insecticide activity and bioefficacy in an assessment. A few experimental hut studies have looked at the impact of insecticide on bed nets with and without holes [6, 7] [10, 54] [11–13, 15–17]; however, the impact of holes of different size, shapes and locations is not clear. More research is needed to evaluate both bioefficacy and physical integrity for LLINs.

There is a programmatic need to develop a standard, reproducible, logistically simple, and comparable method to assess net condition under normal use environments. The PHI provides a standardized approach to estimate the total surface area (or relative area using the weights) damaged by holes on a net. Results of the current detailed analysis using bed nets collected over three years in Mozambique reveal some of the potential weaknesses of the PHI including: underestimating the contribution of really large holes, overestimating the contribution of smaller holes, ignoring extra small holes and hole location, and oversimplifying the hole shape as circular. More research is needed to better refine the PHI and to understand the factors associated with hole initiation and enlargement in field conditions. Information about the size, shape, location and type of holes may help develop a simple field strategy for assessing net usability in terms of physical integrity that will help programs develop plans for repair or replacement of nets. More importantly, further investigation is needed to determine what hole size(s) put net users at the greatest risk in order to determine the optimal time for net replacement.

Bed net monitoring and evaluation (M&E) activities including coverage surveys and malaria indicator surveys are an integral part of national malaria control program activities and provide information on general bed net ownership and use. The PHI is designed to help determine whether a large population of bed nets are still protective or should be replaced; however, in field situations identifying and counting all of the holes in a net, tabulating the data, and calculating a PHI may be laborious and time consuming. Due to the large volume of nets under evaluation, a simpler method of determining the usability of a net based on its physical condition that could be easily incorporated in existing M&E activities would be of great value.

Physical integrity assessments including both in depth analysis of a small sample of nets and simple field counting strategies on larger numbers of nets during household surveys can be used in conjunction with insecticidal activity monitoring to enhance malaria control efforts in the field and can help guide manufacturers to build longer-lasting bed nets.