In this prospective study, we collected data during the first month of the H1N1/09 outbreak between April 28, 2009 and May 28, 2009. We monitored coverage of H1N1/09 influenza in real time on the English-language CDC webpage titled “H1N1 Flu (Swine Flu): Information for Specific Groups” (http://www.cdc.gov/H1N1FLU/update.htm), which classifies the documents according to their intended audience. We captured the full text of all posted guidance documents during that period.

We assessed documents according to one measure of literacy level and one measure of overall readability. In assessing literacy level, we used a Microsoft Word utility to calculate the Flesch-Kincaid (FK) grade level, which yields a score corresponding to a United States school grade reading level. Widely used in studies assessing health information available on the World Wide Web [7],[8],[9],[10],[11], the FK score takes into account the average number of syllables per word and average length of sentences in the document. We chose this measure because of its wide usage and its automated nature, which allowed us to assess all the documents through a uniform procedure.

The FK grade level score is calculated based on verbal content but not other factors (including layout, visual ease of reading, and presentation format) that can determine access and comprehensibility. In order to address this shortcoming, we also used the Suitability Assessment of Materials (SAM) instrument [12] to assess the overall readability of the guidance documents targeting two particular audiences: laypersons and those in secondary distributive institutions (e.g., educational institutions, businesses and employers, and travelers and the travel industry). This subset of CDC guidance documents was chosen for two reasons. First, our initial inspection of these documents indicated that the formatting and wording of documents intended for technical professionals and primary distributive institutions (e.g., state or tribal health departments) did not vary much by topic or across time. However, such variations were observed among the subset of documents targeting laypersons and secondary distributive institutions, suggesting that there had been efforts to tailor the presentation of specific topical information to these audiences. Second, it was thought that document readability would be a more critical predictor of engagement with and comprehension of the target information among laypersons and members of the secondary distributive audience.

Developed by Doak, Doak, and Root for evaluating patient education materials, the SAM score takes into account criteria such as layout, typography, graphics, and surrounding context; thus assessing the entire presentation of the document and its accessibility to an intended audience. Scores for each category range from 0 (Not Suitable) to 2 (Superior). Though the SAM was originally validated in a clinical sample [12] and was developed for and is widely-applied in coding paper documents [13], [14], [15], [16], [17], the instrument has been previously employed in studies assessing the readability of online health education materials [18], [19], [20]. To our knowledge, no modified versions of the SAM have yet been developed to capture the unique experience on reading online. In order to better characterize readability features relevant to the internet, we modified the tool to improve its application to Web-based documents. We removed factors that were directly related to reading on a printed page, such as high contrast between the typeface and the page and the glossiness of the paper. We also added an item capturing the inclusion of World Wide Web links in order to better reflect the online information-seeking experience. With these revisions, the modified SAM assessed the following content-level characteristics: inclusion of a statement of purpose, the scope of material covered, and inclusion of a summary statement. Literacy-related features assessed in the SAM included the reading level (as assessed using the Flesch-Kincaid score), writing style, vocabulary, inclusion of context when introducing new material. Document-level features included the layout and typography as well as the use of subheadings and “road signs” to guide the reader through the document. Lastly, the SAM assessed reader stimulation, including factors related to text interaction with the reader (e.g., the use of questions or cues to action), links to additional Web sources, and reader motivation (e.g., partitioning of subtopics into smaller sections). Two independent coders coded all the content, and discrepancies were resolved through discussion in order to reach 100% coding agreement. A full description of the coding criteria is summarized in Table S1.

We identified the target audience for each document based on its placement on the CDC website (http://www.cdc.gov/H1N1FLU/update.htm). CDC identifies 12 different audiences, some with subgroups: emergency shelters; health care providers; state, local, tribal and territorial health officials; laboratorians; pharmacists; parents and caregivers; educational institutions; community and faith-based organizations; people at high risk for flu complications; businesses and employers; travelers and the travel industry; and people in contact with pigs. We collapsed these groups into six broad categories: technical personnel (health care providers, laboratorians and pharmacists); primary distributive institutions (state, local, tribal, and territorial health officials); secondary distributive institutions (educational institutions, businesses and employers, and travelers and the travel industry); laypersons (parents and caregivers, people at high risk for flu complications and people in contact with pigs); media (documents downloaded from the “press updates” page); and multiple audiences. Materials in this last category consisted of general reports and fact sheets. Within each category, documents were classified further by the date of original release in three time periods: first week after the outbreak, second week after the outbreak, and thereafter.

We proposed three hypotheses. First, we expected the FK grade level scores to decrease with progression through the audience chain, such that the scores would be highest for the technical audiences and lowest for laypersons. This was based on the reasoning that information meant for the consumption of technical audiences would contain more professional or scientific jargon and technical information, as compared to information meant for the consumption of lay audiences. Second, we expected the FK grade level scores to decrease over time. Because translating highly technical language into simpler forms takes more time and effort, we reasoned that, during the initial phase of the outbreak, the urgency to disseminate information quickly would likely outweigh the need to spend more time prioritizing information and refining the language. Over time, however, the language would become simpler as more resources are brought to bear and communication efforts become more refined. Lastly, we expected that there would be a time-by-audience interaction effect, such that reading level would decrease over time more rapidly for audiences further down in the audience chain (e.g., the laypersons and media) than for audiences further up in the audience chain (e.g., technical and primary distributive audiences).

Next, we used the SAM tool to assess the subset of documents targeting secondary distributive institutions and laypersons. We compared the absolute and mean scores for each SAM category, highlighting areas of strength and weakness in the writing and presentation of information for these two distinct audiences. The application of the SAM was intended to provide insight and context for the FK grade level scores.

During the study period, a total of N = 101 unique guidance documents were published on the CDC website and were captured for this analysis. Of these, 22 were identified as targeting technical audiences, 6 as targeting primary distributive institutions, 15 as targeting secondary distributive institutions, 27 as targeting laypersons, 13 as targeting media, and 18 as targeting multiple audiences (including those for whom particular audiences could not be determined).

The average FK grade level score for all documents in our sample was 8.57 (SD = 2.31), corresponding approximately to a ninth-grade reading level. The FK grade level scores across audiences and over time are shown in Figure 1. Across the audience types, the highest grade level score was observed in documents targeting the primary distributive institutions (M = 11.2, SD = .62) and the lowest score across the entire time period was observed in documents targeting the media (M = 6.05, SD = .67). Overall, there was a significant main-effect of audience type, F(5, 82) = 29.72, P<.001. Grade level scores in documents targeting primary distributive institutions did not differ from those targeting technical audiences (M = 10.9, SD = 1.76), but both of these were significantly higher than the grade level scores targeting the other audiences. The grade level scores in documents targeting the media were significantly lower than those targeting the other groups.

Contrary to our expectations, grade level scores did not vary over time: F(2, 82) = .34, P>.05 on the FK grade level scores). There was, however, a significant audience × time interaction effect (F(10, 82) = 2.11, P<.05), the patterns of which are shown graphically in Figure 1. The grade level scores did not follow a discernible pattern across the different groups over time, but a post-hoc analysis revealed that the significant interaction was due to the patterns of means pertaining to laypersons. In particular, grade level scores of guidance documents targeting laypersons during the second week were significantly different from those targeting laypersons during the first week (Z = 3.23, P<.01) and those during the subsequent weeks of the study (Z = 3.53, P<.001). The low grade level score observed for laypersons during the second week was, in fact, significantly lower (all Z's>1.96 and P's<.05) from all other 16 scores, except for two: documents targeting the media during the first week and the second week (for which the scores were not significantly different). The overall ANOVA model testing the effects of audience type, time, and audience × time interaction on FK grade level scores was significant, F(17,82) = 10.25, P<.001.

The SAM tool was used to evaluate the readability of documents targeting secondary distributive institutions and laypersons in order to assess their readability. Scores for each of the readability criteria are presented in Table 1. Among documents targeting secondary distributive institutions (including, for example, local public health departments), three content-level factors emerged as areas of particular strength: the inclusion of a consistent and clearly defined statement of purpose (SAM score = 1.29); limiting the scope of information provided to only that pertaining to the stated purpose (SAM score = 2.0); and the appropriate use of Web links, guiding the reader to additional resources (SAM score = 1.93). One area requiring improvement was the inclusion of a summary statement. None of the documents in this sample included a summary statement to help the reader integrate and act on the information provided in the guidance document. Another area of considerable weakness among documents targeting secondary distributive institutions was the absence of rhetorical techniques that encourage interaction with the reader, such as posing questions and suggesting specific actions (SAM score = 0.07).

Among those factors specifically assessing literacy levels, writing style was found to be acceptable among this sample of documents (SAM score = 0.86), as defined by the scoring criteria developed by Doak, Doak, and Root (1996) [12]. However, the complexity of vocabulary may have been too high in guidance documents targeting secondary distributive institutions (SAM score = 0.43), consistent with the higher reading grade level for this set of documents (FK = 9.5). Lastly, these documents require much improvement in their visual design and layout (SAM score = 0), meaning that they included none of the design elements that tend to improve readability. Similarly, these documents relied heavily on paragraph format, neglecting to break complex information down into more manageable chunks, such as in the form of bulleted lists (SAM score 0.29).

As with documents targeting secondary distributive institutions, two areas of particular strength among guidance documents aimed at the lay audience were the inclusion of a clearly defined statement of purpose (SAM score = 1.53); information presented in the documents was limited in scope consistent with the stated purpose (SAM score = 1.4). Additionally, these documents were strong in their use of advance organizers such as headers and topic captions (SAM score = 1.73), making the text appear more organized and allowing the reader to cognitively orient to the topic ahead. However, other positive aspects of visual presentation on the webpage were lacking. As with documents aimed at secondary distributive institutions, those targeting laypersons scored low on layout and the use of “chunking” (SAM scores = 0.0 and 0.13, respectively). As a result, although the reading level of this subsample of documents was appropriately geared towards a lay audience, the layout and organization of the text may have made these documents relatively difficult to process. Additionally, none of the documents in this subsample included a summary statement recapitulating the key messages, thus reducing the potential for retention and providing relatively weak encouragement for readers to act on the information.

A number of limitations to this study require acknowledgment. First, while some tools exist to assess the usability of Internet documents, there are no resources available to specifically assess their readability. As a result, the research team employed the SAM, a tool developed for use with printed materials. Even so, we considered the SAM to be sufficiently suited to the online reading experience as it is possible that it would be even more difficult to read dense, poorly-formatted material on a small computer screen versus a larger page. Several earlier studies indicate that the material read on a screen is more difficult to understand and process than print material [12]–[14]. Consequently, the SAM provides a conservative estimate of online readability. Furthermore, no data were available on whether documents were read online or printed and read on paper; considering the possibility that documents may have been printed and posted (for instances in schools or workplaces), the SAM is an excellent option for assessing a potential variety of reading experiences.

Beyond inclusion on the CDC website, it is not known if and how information about novel H1N1/09 virus was distributed and to whom. The research team was not able to sample guidance documents that might have been sent directly to target audiences rather than (or in addition to) being posted online. It is possible that such documents may have varied in format, content, and readability from those available online. Nevertheless, given the prominence of the CDC website in conveying health information, we deemed it important to understand the nature of this communication. Further investigation would be needed to examine precisely how documents are distributed. Additional research is also needed to explore how members of the public access and use information in the time of an outbreak, and the extent to which they rely on and trust content received through the Internet compared to other sources.

This study demonstrates an effort on the part of the U.S. public health system to tailor written health information about novel H1N1/09 virus to the needs of specific audiences by adjusting the reading level, vocabulary, and writing style. However, reading level alone does not determine information accessibility. Findings from the SAM assessment of CDC guidance documents about H1N1/09 influenza also demonstrate a lack of attention to visual and layout features that can improve the readability of health information. Public health communication should be crafted to reflect the informational needs as well as the technical capacity of the target audience. Further, the formatting of health communication documents should be formatted to improve accessibility, regardless of the target audience. This includes using a medium to large size font, allowing for adequate white apace on the page, limiting sentence length, and grouping bits of text to allow for easier navigation of the document. Future efforts should be made to ensure that all appropriate readability factors are considered when designing vitally important communication materials following the onset of an outbreak, when urgency and uncertainty are high and information needs are greatest.