As a cornerstone of scientific methodology, transparency is openly expressing all research details and procedures. In systems science and simulation modeling, transparency has been a frequent subject of concern over the last decade (Jalali et al., 2021; Martinez-Moyano, 2012; Rahmandad & Sterman, 2012). The recent and rapid proliferation of COVID-19 models has prompted calls for more transparency in reporting quantitative simulation models (Barton et al., 2020; Jalali et al., 2020). In system dynamics (SD), while substantial attention has been paid to the transparency of quantitative models, where methods and reporting standards have been established (e.g., Rahmandad & Sterman, 2012), there has been less discussion about the transparency of qualitative visualization tools.

Causal loop diagrams (CLDs), with or without a stock and flow structure, are a common and powerful tool for a qualitative and visual explanation of the structure and dynamics of a system, thought experimentation within systems, expansion of mental models, and stakeholder engagement (Baugh Littlejohns et al., 2018; Sterman, 2000). While there has been considerable progress in reporting SD models, a major question remains: We build CLDs to make a problem transparent; but do we make our CLDs transparent?

Transparency in SD matters greatly, regardless of whether a qualitative CLD is transformed into a quantitative simulation model, to build confidence in perceptions of SD research by the greater scientific community. The literature has established best practices for an array of topics related to transparency in qualitative SD research methods (Black, 2013; Black & Andersen, 2012; Kim & Andersen, 2012; Luna-Reyes & Andersen, 2003; Tomoaia-Cotisel et al., 2022). However, a review of the degree that transparency is achieved in CLDs is currently missing.

We aim to address these gaps. Particularly, the twofold objective of the current analysis includes an investigation into the status of transparency of CLDs, followed by the provision of a set of recommendations regarding the transparency of CLD reporting. Note that in our defined scope, we solely focus on basic aspects of reporting transparency. We do not assess other critical quality attributes of CLDs, such as clear link and loop polarities and variable names with discernible sense of direction, among others.

Our review process followed the scoping review methodology (Arksey & O’Malley, 2005) to characterize the degree of transparency of CLD reporting present in a specific set of SD literature. We defined as the set of relevant literature a collection of research articles published between 2018–2021 and containing the presentation of a CLD. Given that System Dynamics Review (SDR) is the flagship journal within the field of SD, we sought to investigate transparency characteristics in SDR articles. Additionally, we reviewed a parallel set of publications in other journals to capture a broader set of literature not limited to one journal. We selected publications separately by journal type to investigate whether a problem (if any) is pervasive in the literature. If papers published in the SDR – where high standards are expected – show deficiencies in transparency, it suggests a broader need for improvement in the wider literature. A detailed description of the search, screening, and inclusion determination steps is provided in the supplementary document.

We evaluated the included studies to determine the degree to which they fulfill three basic transparency criteria. We chose these criteria based on two previous transparency assessments in modeling (Jalali et al., 2021; Jalali et al., 2020), in which over 20 transparency criteria were reviewed. For simplicity, we selected three to apply in the current context because these three criteria are universal across CLDs. While other criteria focus on technical aspects like detailed model equations, such specifics fall outside the scope of CLDs. In other words, we chose them based on our reasoning of essential, minimum elements required for the reader to understand the methods and sources that generated the CLD. The first two referred to the CLD development process, and the third referred to the CLD visualization.

The three assessment criteria were: (1) the article includes a direct statement in plain language that describes the method of how the CLD was generated (i.e., via literature review, interviews, group model building (GMB), authors’ intuition, or combinations of these approaches); (2) even if such a direct statement is given, the degree that the method for the development of model structure is discernable by the reader (i.e., given the full breadth of the information contained in the article text and figures); and (3) the degree of identification of the information source for causal links depicted in the CLD. The first criterion was summarized with a binary yes/no designation. A lack of a clear statement on the CLD development method does not preclude authors from describing their CLD and its sources. Thus, papers could still achieve scores for the second and third criteria despite a “no” for the first. Within the second and third criteria, a score out of 3 possible points was assigned. The scoring rules were defined as: 1=unclear/not described; 2=described with some detail missing; 3=fully identified. Criterion 3 could be satisfied if source identification takes place directly in the visualization of the CLD, in a tabular form, or otherwise discernible based on the text provided. For CLD generation approaches based less on quantitative data or literature, such as those based exclusively on individual inputs, e.g., GMB, citations on individual causal links in the visualization would not apply equivalently. While we note that practical approaches to identify the sources of specific links as attributable to individual inputs are possible, such as via detail in methods text or within an illustration, such cases were considered not applicable for the third criterion for the purpose of our review. Table 1 presents the summary of the three transparency assessment criteria.

After collecting data from the completed extraction sheet, we identified practical recommendations for improving CLD transparency. We based these on two factors: 1) literature sources and 2) our experience, along with observations from our review. The literature offered foundational recommendations to enhance CLD reporting transparency across multiple approaches. Building upon prior work, we categorized core approaches to create CLDs and presented recommendations for each approach. Observed gaps in reporting and other insights gleaned from our assessment of recent CLDs affirmed the need for specification of such transparency guidance and supplemented our recommendation framework. We summarized these recommendations in the Discussion section.

The search strategy and inclusion criteria resulted in 72 articles: 36 from SDR and 36 from other journals (see supplementary Figure S–1). For the first transparency assessment criterion, only 32 articles (44%) directly communicated the methods underlying the CLD structure. Twenty-seven (38%) fully satisfied the second criterion regarding the degree of clarity in communicating the approach driving the CLD structure, receiving a score of 3. Twenty-four (33%) and 21 (29%) received scores of 2 and 1, respectively, for this second criterion. Finally, only 18 (25%) fully satisfied the third criterion regarding the extent to which individual causal link sources in the CLD structural representation were discernable to the reader. Thirty-nine (54%) and 9 (13%) had respective scores of 2 and 1 for this third criterion. Supplementary Table S–1 reports the details of the analysis.

Results separated by articles appearing in SDR versus other journals are displayed in Figures 1–3. Figure 1 shows that both SDR and non-SDR studies had a similar proportion (44%) that omitted a direct method statement explaining the CLD development approach.

As shown in Figure 2, CLDs presented in SDR articles contained higher clarity versus other journal articles in their communication of the process of CLD development. Specifically, 89% of SDR articles received a score of at least 2, compared to 53% in other journals. However, with less than half of the articles receiving the top score of 3, the results demonstrate apparent room for improvement across all literature sets.

Figure 3 illustrates mixed outcomes for the third criterion under evaluation. While CLDs from SDR generally fared slightly better than those from other journals in denoting sources for individual causal links (86% versus 73% achieving a score of at least 2), the difference was more pronounced for those achieving the top score of 3 — 44% in SDR articles compared to 6% in other journals.

An article from the sample we reviewed that is characterized by high scores in our transparency assessment may prove to be an illuminating example when contrasted with methods from lower-scoring articles. Matchar et al. (2018) is one of the reviewed articles that received the highest scores possible across all three criteria. For the first criterion, the CLD development approach is directly identified by Matchar et al. (2018) as coming from a combination of literature and input from experts and stakeholders. The Matchar et al. (2018) methods description employs clear language about the development process in a concise identification of steps to first develop a conceptual model supported by literature, followed by presentation of the conceptual model to experts for verification of structure and assumptions, and finally by implementation of an iterative refinement process supported further by expert input. The article thus satisfies our second transparency criterion concerning the overall clarity of the CLD development method. Meanwhile, an article that scored 1 does not describe the CLD method in any way. An article that scored 2 gives only a partial indication to state that a model has been developed on a particular topic, and while authors cite some literature for a subset of the key relationships, it remains unclear to the reader whether the literature served as a foundation for CLD development or if that literature was used rather to validate relationships represented that may have been constructed using an alternate method.

Finally, to contrast articles in terms of the third and final transparency criterion via specific examples, Matchar et al. (2018) present a CLD visualization where each causal link either directly maps to a specific information source, or they are attributed to expert or stakeholder input as described in the methods text. Note that while the format of producing citations directly in the diagram is one way to satisfy this criterion, other ways such as an adequate description in the text about causal link sources would equally be sufficient for a score of 3.

Additional examples of articles that scored highly across the transparency criteria may prove useful. Rahmandad et al. (2021) present a model to estimate disease transmission using the established Susceptible-Exposed-Infected-Recovered (SEIR) model framework as a foundation. This article scores “yes”, “3”, and “3” in the three transparency assessment criteria. The authors extend the SEIR model and clearly delineate which aspects are the novel model features and that the new features are built around author assumption and literature. Another article that scores “yes”, “3”, and “3” in the three transparency assessment criteria concerns modeling water resource management and also utilized an existing framework (in this case, the Driver-Pressure-State-Impact-Response (DPSIR) model) as a starting place to build an extension (Zare et al., 2019). Authors base the extension on two sources: literature and interviews. They generate the CLD extension in the context of a case study, where links are based on literature sources, and proceed to confirm and refine the CLD based on informal interviews with experts and stakeholders familiar with the problem. Individual links are then able to be traced to cited literature provided in the text and are otherwise attributed to qualitative information gathered in interviews. The limited aspects of these models, which extend beyond the base SEIR and DPSIR models, simplify the explanation of sources. This makes it easier for readers to trace the new parts of the CLDs to the methods description and sources.

The results of our transparency assessment showed that despite our evaluation of highly basic transparency criteria, there remains substantial room for improvement in the current SD modeling literature. Other transparency criteria related to modeling research beyond this baseline level have been the subject of investigation within detailed studies (Jalali et al., 2021; Jalali et al., 2020). Our results show that transparency in publications found in SDR compared to publications in other journals was similar in the provision of clear CLD development approach statement (criterion 1). SDR publications performed better than other journals in their overall clarity of communicating the CLD development approach and causal link sources (criteria 2 and 3), yet they still have deficiencies.

Below we discuss the importance of transparency in reporting CLDs and provide additional examples. We then provide our recommendations for increasing transparency in reporting CLDs.

Our review study documents a deficiency present in the transparency of CLD reporting. Relative to other types of simulation modeling methods, which are sometimes limited to expressions of mathematical equations to communicate the underlying model components, SD models have CLDs available as an invaluable tool to communicate the model structure visually. CLDs are also valuable in their own right, independent of performing a simulation exercise.

Overall, transparent CLDs empower the reader to identify the approach used to generate the CLD—whether that be prior research, individuals’ opinion, group model building, or some combination of these. Improvements in the transparency of CLDs allow a variety of researchers to take full advantage of this tool. Transparency in CLDs as outlined in the recommendations presented in this article can thus help improve the clarity of evidence underlying models and uphold the perception of the field as one built on solid scientific rigor.