pmc10126263532661J Mach Learn ResJ Mach Learn ResJournal of machine learning research : JMLR1532-44351533-79284135584712676262NIHMS2115864ArticleDynamic Bayesian Learning for Spatiotemporal Mechanistic ModelsBanerjeeSudiptoDepartment of Biostatistics, University of California, Los Angeles, Los Angeles, CA 90025, USAChenXiangDepartment of Biostatistics, University of California, Los Angeles, Los Angeles, CA 90025, USAFrankenburgIanDepartment of Biostatistics, University of California, Los Angeles, Los Angeles, CA 90025, USAZhouDanielDepartment of Biostatistics, University of California, Los Angeles, Los Angeles, CA 90025, USA SUDIPTO@UCLA.EDU 410202520250512202526146https://creativecommons.org/licenses/by/4.0/License: CC-BY 4.0, see https://creativecommons.org/licenses/by/4.0/. Attribution requirements are provided at http://jmlr.org/papers/v26/22-0896.html.

We develop an approach for Bayesian learning of spatiotemporal dynamical mechanistic models. Such learning consists of statistical emulation of the mechanistic system that can efficiently interpolate the output of the system from arbitrary inputs. The emulated learner can then be used to train the system from noisy data achieved by melding information from observed data with the emulated mechanistic system. This joint melding of mechanistic systems employ hierarchical state-space models with Gaussian process regression. Assuming the dynamical system is controlled by a finite collection of inputs, Gaussian process regression learns the effect of these parameters through a number of training runs, driving the stochastic innovations of the spatiotemporal state-space component. This enables efficient modeling of the dynamics over space and time. This article details exact inference with analytically accessible posterior distributions in hierarchical matrix-variate Normal and Wishart models in designing the emulator. This step obviates expensive iterative algorithms such as Markov chain Monte Carlo or variational approximations. We also show how emulation is applicable to large-scale emulation by designing a dynamic Bayesian transfer learning framework. Inference on η proceeds using Markov chain Monte Carlo as a post-emulation step using the emulator as a regression component. We demonstrate this framework through solving inverse problems arising in the analysis of ordinary and partial nonlinear differential equations and, in addition, to a black-box computer model generating spatiotemporal dynamics across a graphical model.

Bayesian meldingcomputer modelsstate-space modelsBayesian transfer learningGaussian process regressionmechanistic systemsspatiotemporal analysisuncertainty quantificationIntroduction

Probabilistic learning of spatial-temporal processes governed by physical or mechanistic systems is of interest in a variety of scientific disciplines. Such models are sometimes represented through a system of equations derived from physical or scientific principles, or they may be arbitrarily complex computer programs that simulate a phenomenon. These models are broadly referred to as computer models1 and, at least in the scope of this paper, are deterministic in the sense that rerunning the model with a specific set of inputs always produce identical value of the outputs. In what is considered a seminal manuscript in statistical science, Sacks et al. (1989) describes a framework for modeling the output as a realization of a stochastic process and laying the statistical foundations for designing computer experiments for efficient prediction. This field has blossomed into a substantial area within machine learning and statistical science with the incorporation of ideas from signal processing, dynamical systems, inverse problems, functional data analysis, non-parametric models, Gaussian processes, spatial statistics and several other fields (while a comprehensive review is not the aim of this article, we refer to Santner et al., 2019; Fang et al., 2005; Kang and Deng, 2020, and references therein for diverse perspectives).

Analyzing data from computer experiments refer to interpolating or predicting the output at new inputs after training the model using runs of the computer model without assuming mathematical tractability of the computer model. This is referred to as emulation (see Sacks et al., 1989; Conti and O’Hagan, 2010, with the latter offering an excellent perspective of relative recent literature) and comprises the construction of a statistical model that mimics the behavior of the computer model. Rather than assuming an explicit functional relationship between the inputs and outputs, a Gaussian process is used as a prior on the unknown function (see, e.g., O’Hagan, 1992; Haylock and O’Hagan, 1996; Rasmussen and Williams, 2005) and the outputs are assumed to be a realization of this process.

A second problem of interest is to learn about the parameters in the mechanistic system from field observations while also accounting for the information available from runs of the computer experiment. This is related to the problem of calibration using field data (see, e.g., Kennedy and O’Hagan, 2001; Oakley and O’Hagan, 2004; Higdon et al., 2004; Bayarri et al., 2007a; Higdon et al., 2008; Bayarri et al., 2009, for calibration using field observations in diverse contexts) and can be regarded as an inverse problem seeking optimal values of unknown parameters to be learned from observational field data. We depart somewhat from the paradigm of traditional calibration, where an “optimal” (perhaps unknown) value of the input is assumed. Instead pursue melding (Poole and Raftery, 2000; Gel et al., 2004) and synthesize information from mechanistic systems with field observations. Emulation and calibration becomes challenging when the computer model or posited functional relationship is complex in nature or expensive to compute and any practicable learning framework must account for the scale of the problem.

We focus upon emulating and inferring on spatial-temporal mechanistic systems using a Bayesian hierarchical modeling framework. Figure 1 depicts emulation and calibration in our melding context. A hierarchical description of a physical process consists of a layered approach, whereby simpler conditional dependence structures specify complex relationships. Probabilistic learning proceeds from the posterior distribution given by [process,parameters|data]∝[data|process,parameters]×[process|parameters]×[parameters].

At the top level, the probability model specifies the distribution of the data conditional on the physical process and any other parameters needed to describe the data generating mechanism. The next level represents the underlying mechanistic system as a realization of a stochastic process capturing physical or mechanistic knowledge (as promulgated by Sacks et al., 1989). The last level models uncertainty about parameters.

Probabilistic machine learning using (1) adopts Bayesian inference of all unknown quantities given the data available to the modeler. Bayesian inference for deterministic systems has been developed and explored in settings that resemble computer experiment settings. Examples, by no means exhaustive, include Bayesian melding developed by Poole and Raftery (2000) that pursues inference from deterministic simulation methods and has witnessed substantial use in climate modeling (see Gel et al., 2004, for one application in forecasting problems), Bayesian data assimilation (Wikle and Berliner, 2007) and Bayesian state space models formed by finite difference approximations of differential equations (see, e.g., Wikle, 2003; Stroud et al., 2010; Wikle and Hooten, 2010; Abdalla et al., 2020, for applications encompassing ecology, climate and industrial hygiene).

In this manuscript we specifically focus upon spatial-temporal mechanistic systems and builds upon a rapidly evolving literature in Bayesian learning for computer models (see Liu and West, 2009; Farah et al., 2014; Conti and O’Hagan, 2010; Gu and Berger, 2016; Gu et al., 2019; Gu and Shen, 2020; Gu and Li, 2022; Gu, 2022, and references therein). Notably, Gu et al. (2019) discusses software development and fast implementation of the methodology in Gu and Berger (2016) using experiments and field observations for vector-valued outputs while also focusing upon accelerating computations. Other work on combining Gaussian processes and state-space models may be found in the machine learning literature (see, e.g., Turner et al., 2010; Eleftheriadis et al., 2017), where the process is used to model the transition function rather than stochastic innovations.

Following Conti and O’Hagan (2010) and Gu and Shen (2020), we use matrix-variate distributions with rows and columns corresponding to inputs and locations, but we incorporate temporal emulation over discrete epochs leading to matrix-variate Bayesian dynamic models. In this regard, we differ from Gu and Shen (2020) who use continuous time Kalman filters. While sacrificing some richness in statistical inference, we aim to harness exact analytically tractable distribution theory to generate exact posterior samples without resorting to iterative algorithms such as Markov chain Monte Carlo (MCMC, Robert and Casella, 2005; Gamerman and Lopes, 2006), Variational Bayes (Ren et al., 2011; Blei et al., 2017) or Laplace approximations (Rue et al., 2009). We devise a matrix-variate Forward Filter Backward Sampling (FFBS) algorithm (Carter and Kohn, 1994; Frühwirth-Schnatter, 2006) to emulate dynamically evolving spatial random fields using exact sampling from matrix-normal and Wishart families. Closed form expressions for the necessary statistical distributions (also using hyper-T distributions) are derived as are expressions for log point-wise predictive or posterior predictive distributions that are used for model selection.

For scaling up emulation, we adapt the FFBS algorithm to Bayesian transfer learning for emulating dynamically evolving mechanistic fields with very large amounts of spatial locations or time points. Here, too, we depart from much of the statistical literature on high-dimensional inference with computer models (Higdon et al., 2008; Gu and Berger, 2016; Gu and Li, 2022) where dimension-reduction is achieved using low-dimensional projections embedded within the statistical models (such as orthogonal latent factor models). Our approach here resembles data partitioning strategies such as treed Gaussian processes (Gramacy and Lee, 2008), meta-kriging (Guhaniyogi and Banerjee, 2018; Guhaniyogi et al., 2017) and predictive stacking over data subsets (Presicce and Banerjee, 2024), but is simpler than in geostatistical settings because the dynamic emulator can be designed executed over regular spatial coordinates and time points to enable sequential updating of the FFBS algorithm over the subsets of the data. For probabilistic learning of the model inputs, we regress the observed field on the emulated field and the mechanistic system parameters are estimated with uncertainty quantification using MCMC but without requiring any additional emulation (using “modularized” inference as described in Bayarri et al., 2007a,b, 2009).

The structure of this article is as follows. Section 2 develops a conjugate family of Bayesian state space models for space-time mechanistic systems emphasizing exact conjugate matrix-variate models to emulate the mechanistic system at arbitrary inputs. Section 3 provides Bayesian model comparison metrics to compare different models for emulating the mechanistic system. Section 4 devises the Bayesian transfer learning approach for scaling up emulation over large spatial fields. Section 6 develops probabilistic learning of mechanistic parameters using field-data and Section 5. Section 7 provide a set of illustrations showing the applications of our dynamic framework using diverse mechanistic systems.

Bayesian State-Space Models for Mechanistic Systems

We offer a brief review of Bayesian state-space models (SSMs) – also known as Bayesian dynamic models (DLM) – closely following developments in West and Harrison (1997) and Petris et al. (2009), but adapting or extending familiar distribution theory in these texts to a general multivariate DLM context for our subsequent developments.

Exact (multivariate) Bayesian inference

We consider mechanistic systems depending on space and time. Examples include broad classes of space-time partial differential equations and, even more generally, all deterministic computer models or agent-based models whose inputs include space and time. We build dynamic emulator models that will allow us to introduce spatial dependence in the output of the mechanistic system. Let 𝒳=x1,…,xN be a set of N mechanistic system inputs, where each xi is a d-dimensional vector. Let 𝒮=s1,…,sS denote the set of observed spatial locations. For any combination (x,s)∈𝒳×𝒮, we denote yt(x,s) as the output of the mechanistic system at time point t.

A conjugate Matrix-Normal-Inverse-Wishart Bayesian model for space-time settings is Yt=FtΘt+𝓔t,𝓔t∼ind.ℳ𝒩N×SO,Vt,Σ,t=1,2,…,TΘt=GtΘt−1+Γt,Γt∼ind.ℳ𝒩p×SO,Wt,Σ,t=1,2,…,TΘ0∣Σ∼ℳ𝒩p×Sm0,M0,Σ,Σ∼ℐ𝒲n0,D0, where Yt is N×S whose (i,j)-th element is ytxi,sj,Ft and Gt are N×p and p×p, respectively, with entries completely known, Θt is p×S comprising latent effects over space and Vt and Wt are correlation matrices of order N×N and p×p, respectively, for each t=1,2,…,T. The important distinction from the more conspicuous univariate DLM setting is that 𝓔t and Γt are now matrix-variate normal distributions with zero matrices as their means, Vt and Wt are known correlation matrices modeling dependence across the rows for the respective distributions, and Σ is an S×S covariance matrix modeling the spatial dependence among the columns in 𝓔t and in Γt. The joint prior density pΘ0,Σ in (2) is a Matrix-Normal-Inverse-Wishart, denoted by ℳ𝒩ℐ𝒲Θ0,Σ∣m0,M0,n0,D0, with density function pΘ0,Σ∝detD0n0/2exp−12trD0Σ−12n0S/2ΓSn02(det(Σ))n0+S+12⏟p(Σ)=ℐ𝒲Σ∣n0,D0×exp−12trΘ0−m0⊤M0−1Θ0−m0Σ−1(2π)pS/2detM0S/2(det(Σ))p/2⏟pΘ0∣Σ=ℳ𝒩Θ0∣m0,M0,Σ,

Sampling from matrix normal distribution

1:Input: p×S mean matrix m,p×p lower-triangular Cholesky factor LM of M and S×S lower-triangular Cholesky factor LΣ of Σ
2:Output: Sample from ℳ𝒩p×S(m,M,Σ)
3:function SampleMatrixNormalm,LM,LΣ
4: Draw Z∼ℳ𝒩p×SO,Ip,IS▷ I is identity matrix, O(pS)
5: 5:Θ=m+LMZLΣ⊤▷ OpS2+p2S
6: return Θ
7:end function▷ OpS2
where ΓS is the S-variate multivariate Gamma function and tr(⋅) is the trace function of a matrix. The matrix-normal density of a random matrix corresponds to a multivariate normal density for the vectorized columns of the matrix. For example, the above density pΘ0∣Σ in (3) is equivalent to vecΘ0∣Σ∼𝒩pSvecm0,Σ⊗M0, where vecΘ0 is the pS×1 vector of the stacked columns (first to last) of Θ0 and ⊗ is the Kronecker product (see, e.g., Banerjee and Roy, 2014, for properties of the vec operator and its connections to the Kronecker product). Standard calculations for multivariate normal distributions yield pΘt,Σ∣Y1:t=ℐ𝒲Σ∣nt,Dt⏟pΣ∣Y1:t×ℳ𝒩Θt∣mt,Mt,Σ⏟pΘt∣Y1:t,Σ=ℳ𝒩ℐ𝒲Θt,Σ∣mt,Mt,nt,Dt, where mt,Mt,nt and Dt can be calculated recursively over t as we describe shortly. Sampling Θt,Σ∼ℳ𝒩ℐ𝒲mt,Mt,nt,Dt proceeds by first drawing Σ∼ℐ𝒲nt,Dt and then drawing Θt∼ℳ𝒩mt,Mt,Σ using the drawn value of Σ.

Algorithm 1 describes the procedure to sample from the matrix-normal distribution using, as input, the mean and the lower-triangular Cholesky factors of the two covariance matrices. We indicate the complexity in terms of the number of floating point operations (flops) for each step in the right-hand column of Algorithm 1. Drawing Z∼ℳ𝒩p×SO,Ip,IS in Line 4 costs∼O(pS), while Line 5 costs∼OpS2+p2S, so the total cost is OpS+pS2+p2S∼OpS2 since p≤S in our subsequent applications.

Closely related to equations (3) and (4) is the Hyper-T distribution. We obtain it by integrating Σ from the Matrix-Normal-Inverse-Wishart (Gupta and Nagar, 1999; Zhu et al., 2007). Doing so for, say, equation (4), which we denote by ℋ𝒯Θt∣mt,Mt,nt,Dt, gives us: pΘt∣Y1:t=∫Σ>0 ℳ𝒩ℐ𝒲Θt,Σ∣mt,Mt,nt,DtdΣ=ΓSnt+p2(det(IS+Dt−1Θ−mt⊤Mt−1Θ−mt))−nt+p2πpS/2ΓSnt2detMtS/2detDtp/2=ℋ𝒯Θt∣mt,Mt,nt,Dt

Sampling from hyper-T distribution

1:Input: p×S matrix m, row-covariance matrix p×p matrix M, positive integer degrees of freedom n, positive symmetric definite S×S scale matrix D
2:Output: Sample from ℋ𝒯p×S(m,M,n,D)
3:function SampleHyperT(m,M,n,D)
4: Draw Σ∼ℐ𝒲S×S(n,D)▷ OS3
5: LM←Cholesky(M)▷ Op3
6: LΣ←Cholesky(Σ)▷ OS3
7: Θ←SampleMatrixNormalm,LM,LΣ▷ Algorithm 1
8: return Θ
9:end function▷ OS3

The canonical form given in (Gupta and Nagar, 1999) or (Zhu et al., 2007) can be obtained by setting the degrees of freedom for the inverse-Wishart nt←nt+S−1, implying that the hyper-T is the matrix-T distribution with degrees of freedom nt−S+1. (It is thus important to point out that the degrees of freedom of the inverse-Wishart nt is not automatically equal to the degrees of freedom for the canonical form of the matrix-T.) Each entry of the matrix-T random variable is a univariate t-distribution with degrees of freedom nt−S+1, mean mt,ij, and scale term Dt,iiMt,jj/nt.

Algorithm 2 describes the procedure to sample from the ℋ𝒯p×S(m,M,n,D). As in Algorithm 1, the right-hand side column provide the complexity in flops. Lines-4, 5, and 6 together cost∼Op3+2S3 and Line-7 costs∼OpS2, which brings the total cost to ∼Op3+2S3+pS2∼OS3 since p≤S in our subsequent applications.

Also important to consider is a matrix-normal Θt, but where Σ=σ2R, where σ2 is distributed as inverse-gamma (ℐ𝒢nt,dt) and R being a given matrix. We denote this density by ℳ𝒩ℐ𝒢Θt,σ2∣mt,Mt,nt,dt,R. Next, we marginalize out the σ2 to obtain a multivariate t-distribution, but one in which the mean arguments are matrices rather than vectors. We call this the hyper-T-scalar and denote it by ℋ𝒯sΘt∣mt,Mt,nt,dt,R: pΘt∣Y1:t,R=∫σ2>0ℳ𝒩ℐ𝒢Θt,σ2∣mt,Mt,nt,dt,Rdσ2=ΓpS2+ntΓ(nt)2πdt−pS2Mt−S2|R|−p2×1+12dttrR−1Θt−mt⊤Mt−1Θt−mt−pS2+nt=ℋ𝒯sΘt∣mt,Mt,nt,dt,R

The matrix-variate SSM in (2) delivers posterior and predictive distributions in closed form. Exact Bayesian inference is possible by sampling from the posterior distribution pΘ0:T,Σ∣Y1:T using the Forward Filter Backward Sampling algorithm (Carter and Kohn, 1994; Frühwirth-Schnatter, 2006). The basic idea is fairly simple. We first move forward in time up to t=T and draw one instance of (Σ,ΘT) from pΘT,Σ∣Y1:T. This is the “forward filtering” step (“FF”). Next, we execute “backward sampling” (BS): for each t=T−1,T−2,…,0, we draw one instance of Θt from pΘt∣Σ,Θt+1,Y1:t. This entire process yields one instance of (Σ,Θ0:T) from pΘ0:T,Σ∣Y1:T. Backward sampling makes use of the Markovian structure in (2), which implies that pΘ0:T,Σ∣Y1:T=pΘT,Σ∣Y1:T∏t=0T−1pΘt∣Σ,Θt+1,Y1:t.

Kalman (forward) filter

1:Input: Data Y1:T, hyperparameters n0,D0,m0,M0, correlation matrices V1:T, W1:T, observation and state transition matrices F1:T and G1:T.
2:
3:Output: Filtering distribution parameters at time t=0,…,T
4:function FilterY1:T,n0,D0,m0,M0,G1:T,F1:T,V1:T,W1:T
5: for t=1 to T do
6:  # Compute prior distribution Θt,Σt∣Y1:t−1∼ℳ𝒩ℐ𝒲(at,At,nt*,Dt*) :
7:  at←Gtmt−1,At←GtMt−1Gt⊤+Wt,nt*←nt−1,Dt*←Dt−1▷ Op3+p2S
8:  # Compute one-step-ahead forecast pYt∣Y1:t−1∼ℋ𝒯(Ftat,FtAtFt+Vt,nt*,Dt*)
9:  qt←Ftat,Qt←FtAtFt⊤+Vt▷ OpSN+p2N+N2
10:  # Compute filtering distribution pΘt,Σt∣Y1:t∼ℳ𝒩ℐ𝒲mt,Mt,nt,Dt :
11:  mt←at+AtFt⊤Qt−1Yt−qt,Mt←At−AtFt⊤Qt−1FtAt⊤
12:▷ OpN2+p2N+p3+pSN+N3
13:  nt←nt*+N,Dt←Dt*+Yt−qt⊤Qt−1Yt−qt▷ OSN2+S2N
14: end for
15: return nt,Dt,at,At,mt,Mtt=0T
16:end function▷ OTp2S+S2N

The “FF” followed by the “BS” steps complete one iteration of the FFBS algorithm yielding one draw from pΘ0:T,Σ∣Y1:T. In essence, the FFBS algorithm first applies the Kalman filter of Algorithm 3 before initializing the backward sampling through the Kalman or Rauch–Tung–Striebel smoother (Rauch et al., 1965; Särkkä and Svensson, 2013). Repeating the FFBS cycle L times yields L posterior samples of Θ0:T and Σ. In object-oriented computing paradigms, we can execute L draws of Σ,ΘT at the end of FF. Then, for each drawn (Σ,ΘT) we execute T−1 draws of the BS to obtain L posterior samples.

The specific FFBS algorithm to estimate (2) accomplishes the forward filtering step by recursively computing the following quantities for each t=1,…,T, at=Gtmt−1;At=GtMt−1Gt⊤+Wt;qt=Ftat;Qt=FtAtFt⊤+Vt;mt=at+AtFt⊤Qt−1Yt−qt;Mt=At−AtFt⊤Qt−1FtAt;nt=nt−1+N;andDt=Dt−1+Yt−qt⊤Qt−1Yt−qt

Algorithm 3 outlines the steps for forward filtering. The right-hand column provides the computational complexity in terms of flops. Lines-7, 9, 11, and 13 together complete one iteration of the for-loop, with a combined cost of ∼Op3+p2(S+N)+pSN+N2+S2N+SN2+N3. In particular, if p,N≤S, then the combined cost is ∼Op2S+S2N for each t. This yields a total cost of ∼OTp2S+S2N. We provide the costs in our subsequent algorithms using p,N≤S. This is true for all our applications except the predator-prey system in Section 5.1.

Backward sampler

1:Input: Filtering parameters and inputs from Algorithm 3
2:Output: L posterior samples from pΘ0:T,Σ∣Y1:T
3:function BackwardSample(nt,Dt,at,At,mt,Mt,Gtt=0T,L)
4: Draw L samples from Σ∼ℐ𝒲nT,DT▷ OLS3
5: Draw L samples from ΘT∼ℳ𝒩mT,MT,Σ▷ OLS3
6: hT←mT,HT←MT
7: for t=T−1 to 1 do
8:  ht←mt+MtGt+1⊤At+1−1ht+1−at+1▷ Op3+p2S
9:  Ht←Mt−MtGt+1⊤At+1−1At+1−Ht+1At+1−1Gt+1Mt▷ Op3
10:  Draw L samples from Θt∼ℳ𝒩ht,Ht,Σ▷ OLS3
11: end for
12: return h1:T,H1:T and L samples of Θ0:T,Σ
13:end function▷ OTLS3

We then draw one instance of ΘT,Σ∣Y1:T∼ℳ𝒩ℐ𝒲mT,MT,nT,DT*, where at and mt are both p×S and qt is N×S. For the inverse-Wishart parameters, nt=nt−1+N is a scalar and Dt=Dt−1+Yt−qt⊤Qt−1Yt−qt is S×S. An exact FFBS algorithm proceeds by forward filtering to t=T to draw L samples from pΘT,Σ∣Y1:T from (4). Next, for backward sampling we exploit standard Bayesian probability calculations to note that Θt,Σ∣Y1:T∼ℳ𝒩ℐ𝒲ht,Ht,nT,DT, where hT=mT,HT=MT, and ht=mt+MtGt+1⊤At+1−1ht+1−at+1andHt=Mt−MtGt+1⊤At+1−1At+1−Ht+1At+1−1Gt+1Mt.

Therefore, for each t=T−1,…,0 we draw one value of Θt∼ℳ𝒩ht,Ht,Σ for each of the L values of Σ at the end of the “FF” step, where ht and Ht are computed using (9). The end of the FFBS yields L posterior samples from pΘ0:T,Σ∣Y1:T.

Forward-filter-backward-sampler algorithm

1:Input: Data Y1:T and Kalman filter starting values n0,D0,m0,M0
2:  Observation and state transition matrices F1:T and G1:T
3:  Correlation matrices V1:T and W1:T, number of samples L
4:Output: Sample from posterior pΘ0:T,Σ∣Y1:T
5:function FFBSY1:T,n0,D0,m0,M0,F1:T,G1:T,V1:T,W1:T,L
6: FFOut←FilterY1:T,n0,D0,m0,M0,G1:T,F1:T,V1:T,W1:T▷ Algorithm 3
7: BSOut←BackwardSample(nt,Dt,at,At,mt,Mt,Gtt=0T,L)▷ Algorithm 4
8: return BSOut={h1:T,H1:T,{Θ0:T(l),Σ(l)}l=1L}
9:end function▷ OTLS3

Algorithm 4 outlines the steps for backward sampling, with the computational complexity detailed in the right-hand column in terms of flops. Lines-4 and 5 each draw L samples of Σ and ΘT, respectively, with a computational cost of ∼OLS3 for both. Meanwhile, Lines-8, 9, and 10 together constitute one iteration of the for-loop, with a cost of ∼Op3+p2S+LS3∼OLS3. This results in a total cost of ∼OTLS3 assuming p,N≤S. Algorithm 5 presents the complete forward-filter-backward-sampler algorithm, incorporating both Algorithm 3 and Algorithm 4. The overall computational cost is ∼OTp2S+S2N+LS3∼OTLS3.

Bayesian predictive inference is also convenient if we seek to predict or impute the values in a new N˜×S matrix Y˜t corresponding to a new set of mechanistic inputs 𝒳˜, given the N˜×p matrix F˜t for any given t, where N˜ represents the number of new mechanistic system inputs. The predictive distribution for Y˜t follows from the augmented model, YtY˜t=FtF˜tΘt+𝓔t𝓔˜t,𝓔t𝓔˜t∼ind.ℳ𝒩(N+N˜)×SON×SON˜×S,VtJtJt⊤V˜t,Σ, for t=1,2,…,T. Therefore, for each drawn value of Θt,Σ∼pΘ0:T,Σ∣Y1:T from the FFBS algorithm, we draw one instance of Y˜t from the conditional predictive density p(Y˜t∣Y1:T,Θt,Σ)=ℳ𝒩(Y˜t∣F˜tΘt+Jt⊤Vt−1Yt−FtΘt,V˜t−Jt⊤Vt−1Jt,Σ).

Repeating this for all the posterior samples of Θ0:T,Σ will yield samples of Y˜1:T from the posterior predictive distribution pY˜1:T∣Y1:T. Therefore, predictive inference can be carried out using (11) for arbitrary inputs in 𝒳˜ using stored posterior samples from the training data. The full expression for the posterior predictive density is itself a Hyper-T, with the following expression: p(Y˜t∣Y1:T)=ℋ𝒯Y˜t∣F˜tht+Jt⊤Vt−1Yt−Ftht,F˜tHtF˜t⊤+V˜t−(FtHtF˜t⊤+Jt)⊤(FtHtFt⊤+Vt)−1(FtHtF˜t+Jt),nT,DT)

The same conclusions we draw as for equation (5) lead us to conclude that each entry of Y˜t,Y˜t,ij, is a univariate-t, with mean F˜t,iht,:j+Jt,:i⊤Vt−1Yt,:j−Ftht,:j and scale parameter (F˜t,j:HtF˜t,j:⊤+V˜t,jj−(FtHtF˜t,:j⊤+Jt,:j)⊤(FtHtFt⊤+Vt)(FtHtF˜t,j:⊤+Jt,:j))(DT,ii)/nT.

We do not require fresh training of the model for predictive inference provided the distribution in (10) is valid—a matter we turn to next.

Mechanistic emulation using Gaussian Processes

Emulating the mechanistic system requires that we train the model in (2) from the inputs in 𝒳 using the FFBS algorithm and, subsequently, to predict or interpolate the outcome matrix for possibly new inputs in 𝒳˜. For predictive inference, we must ensure that the distribution in (10) is well-defined, which, in turn, requires that VtJtJtV˜t is positive definite for arbitrary mechanistic inputs 𝒳∪𝒳˜. This will be ensured if we ensure that each column of the augmented outcome matrix YtY˜t is a realization of a valid stochastic process over 𝒳∪𝒳˜. A customary choice for mechanistic emulation is the Gaussian process.

The Gaussian process acts as a prior on of YtY˜t. Thus, for each of the S locations (or column indices), we assume yt(x,s)∼ind𝒢𝒫μt(x,s),C(⋅,⋅;β), where the mean function μt(x,s)=ft(x,s)⊤Θt(s) and a positive definite correlation function Cx,x′;β. This yields, Vt=Cxi,xj;β for pairs of inputs in 𝒳,V˜t=Cx˜i,x˜j;β for pairs of inputs in 𝒳˜ and Jt=Cxi,x˜j;β for xi∈𝒳 and x˜j in 𝒳˜. This specification ensures that (10) is a valid probability distribution by virtue of Cx,x′;β being a valid correlation function. Several options for valid correlation functions are available, including the rich Matérn class used widely in spatial statistics. In mechanistic emulation, we are mostly concerned with interpolation so any valid correlation function of a simpler form will suffice. For our current illustrations, we use the squared exponential function Cx,x′;β=exp−∑i=1dβixi−xi′2, where the range parameter βi controls the decay of correlation along the i-th dimension of β. In particular, let us denote Yt and Y˜t as Yt(𝒳) and Y˜t(𝒳˜), respectively. Then, the predictive mean EYt(𝒳˜)∣Yt(𝒳),Θ0:T,Σ in (11) acts as an interpolator over the mechanistic inputs. More precisely, if 𝒳˜⊆𝒳, then the rows of Jt⊤ are a subset of the rows of Vt, which implies that Jt⊤Vt−1=I(𝒳˜) (since VtVt−1=I(𝒳)), where I(𝒳˜) is the subset of rows of the identity matrix indexed by 𝒳˜. Therefore, Jt⊤Vt−1Yt−FtΘt=Yt(𝒳˜)−F˜tΘt and E[Yt(𝒳˜)∣Yt(𝒳),Θ0:T,Σ]=F˜tΘt+Jt⊤Vt−1Yt−FtΘt=F˜tΘt+Yt(𝒳˜)−F˜tΘt=Yt(𝒳˜).

Furthermore, this interpolation is deterministic because Var[Yt(𝒳˜)∣Yt(𝒳),Θ0:T,Σ]=V˜t−Jt⊤Vt−1Jt=V˜t−I(𝒳˜)Jt=V˜t−V˜t=O, where the second to last equation follows from the fact that I(𝒳˜)Jt=V˜t when 𝒳˜⊆𝒳.

Limitations

If we are able to evaluate and store the output from a space-time mechanistic system over a fixed set of S locations and T time points, then we are able to learn about the temporal evolution of the process and the association over the spatial locations (through Σ) using a conjugate Matrix-variate dynamic linear model framework. A clear advantage of this approach is that inference from the FFBS samples from the exact posterior and posterior predictive distributions and there is no need to resort to more computationally expensive iterative algorithms (e.g., MCMC, INLA, VB) that will require diagnosis of convergence.

The relative simplicity of this learning framework, however, comes with some severe limitations with respect to spatial modeling. First, and perhaps most obviously, the above approach uses an unstructured Σ to model the associations across space, whereas it could be argued that one should model Σ further using conventional geostatistical kernels that more explicitly model association as a function of the locations (or distances between them). Second, since the dimension of Σ is fixed, learning of the mechanistic system occurs only over the fixed set of S spatial locations that have been determined at the design stage to run the system but precludes predictions at arbitrary new locations. Third, it does not accommodate the possibility that for each spatial location, we may not have the same inputs (breaking the N×S matrix structure) which is the likely scenario when our learning framework includes information from field data that we use to calibrate the mechanistic system.

A more flexible approach builds a learning system that models both the mechanistic system and the spatial associations using stochastic processes with Gaussian processes being the customary choice here. Now Σ is constructed using a spatial covariance kernel which leads to a sparser parametrization, easier interpretation, and accommodates predictions over space. While the exact distribution theory for the S×S covariance matrix Σ is lost, we arrive at a richer and more flexible framework that also eliminates the requirement for the dimension of Yt to be the same for each t. Therefore, we construct Yt=FtΘt+𝓔t,𝓔t∼ind.∼ℳ𝒩Nt×SO,Vt,σ2R,Θt=GtΘt−1+Γt,Γt∼ind.ℳ𝒩pt×SO,Wt,σ2R,Θ0∣σ2∼ℳ𝒩pt×Sm0,M0,σ2R,σ2∼ℐ𝒢n0,d0, where Yt and Θt are Nt×S and pt×S, respectively, Ft and Gt are Nt×pt and pt×pt−1, respectively, R is S×S spatial correlation matrices whose (i,j)-th elements are values of some spatial correlation function ρtsi,sj with fixed process parameters. Fixing Vt,Wt, and R in (14) will, again, yield conjugate Bayesian posterior distributions obtained using an FFBS algorithm.

The FFBS algorithm to estimate (14) adapts Algorithms 3, 4 and 5 by recursively computing the following quantities for each t=1,…,T, at=Gtmt−1;At=GtMt−1Gt⊤+Wt;qt=Ftat;Qt=FtAtFt⊤+Vt;mt=at+AtFt⊤Qt−1Yt−qt;Mt=At−AtFt⊤Qt−1FtAt;nt=nt−1+NS2;anddt=dt−1+12trYt−qt⊤Qt−1Yt−qtR−1.

For the inverse-Gamma parameters, nt is the shape and dt is the rate. As earlier, we use forward filtering up to t=T to draw L samples from pΘT,σ2∣Y1:T from ℳ𝒩ℐ𝒢mT,MT,nT,dT, where at and mt are both pt×S and qt is Nt×S. For backward sampling, we note that Θt,σ2∣Y1:T∼ℳ𝒩ℐ𝒢ht,Ht,nT,dT, where hT=mT,HT=MT, and ht=mt+MtGt+1⊤At+1−1ht+1−at+1andHt=Mt−MtGt+1⊤At+1−1At+1−Ht+1At+1−1Gt+1Mt.

Therefore, for each t=T−1,…,0 we draw one value of Θt∼ℳ𝒩ht,Ht,σ2R for each of the L values of σ2 at the end of the “FF” step, where ht and Ht are computed using (16). The end of the FFBS yields L posterior samples from pΘ0:T,σ2∣Y1:T.

Bayesian Model ComparisonsWidely applicable information criteria

We evaluate predictive accuracy of our models using the Watanabe-Akaike (Widely Applicable) Information Criteria (WAIC) defined as WAIC=−2lppd−pWAIC where lppd=∑t=1TlogEΘt∣Y1:T[p(Yt*∣Θt)] and pWAIC=∑t=1TVarΘt∣Y1:T[logp(Yt*∣Θt)], where we use Yt* for the posterior predictive density p(Yt*∣Y1:T). The “lppd” is the “log pointwise predictive density” measuring how well the model fits the data, while pWAIC is the sum of the posterior variance of the log-predictive density and estimates the effective number of parameters and serves as a penalty for the model. The difference between lppd and pWAIC is multiplied by −2 to be on the deviance scale (Vehtari et al., 2017; Chan-Golston et al., 2020). Hence, lower WAIC values indicate preferred models.

Since our densities are predominantly matrix-normal, with Σ∼ℐ𝒲nT,DT and Σ=σ2R being integrable from the ℳ𝒩ℐ𝒲 and ℳ𝒩ℐ𝒢 respectively, the lppd can be computed analytically for each. Without loss of generality, we address the case with Σ∼ℐ𝒲nT,DT, with moments computed with Algorithm 5: EΘt∣Y1:T[p(Yt*∣Θt)]=p(Yt*∣Y1:T)=ℋ𝒯(Yt*∣Ftht,FtHtFt⊤+Vt,nT,DT). The log-density follows from the expression for the Hyper-T in equation (5): lppd=−NST2logπ+∑t=1TlogΓSnT+N2−logΓSnT2−S2logdetFtHtFt⊤+Vt−N2logdetDT−nT+N2logdetIS+DT−1Yt−Fht⊤FtHtFt⊤+Vt−1Yt−Ftht

Sampling is required to compute the pWAIC, with L samples of Θt∣Y1:T taken to get the empirical pWAIC:pˆWAIC=∑t=1TVarlogpYt*∣Θt(l)l=1,…,L.

Patterning after (Vehtari et al., 2017), we compute the standard error of the WAIC by taking the sample variance across time. Denote the term-wise WAIC by the following: WAICt=−2logEΘt∣Y1:T[p(Yt*∣Θt)]−VarΘt∣Y1:T[logp(Yt*∣Θt)]

Then the standard error of the WAIC can be computed as follows: se(WAIC)=TVar(WAICtt=1,…,T)

Posterior predictive loss criteria

This criteria based on (Gelfand and Ghosh, 1998) involves sampling independent replicates of the model outcome based on the distribution’s computed moments. Denote one such set of replicates by Yt(rep), where Yt(rep) is a “future” observation which is replicated from the distribution of Yt. However, since we account for the actual observations Y1,…,T in our criteria, the replicate we sample is actually Yt(rep)∣Y1,…,T. For L such replicates, we sample {Yt(rep),l}l=1,…,L. Then the sample mean of the L samples is μˆt(rep),L=1L∑l=1LYt(rep),l and its per-coordinate variance is σˆt2xi,sj=Var((ytxi,sj(rep),l)l=1,…,L). These two moments are used to define a D-score Gelfand and Ghosh (1998) as the sum of a goodness of fit measure (G) and a penalization term (P). Here, D=G+P, where G and P are defined as G=∑t∑i,j(ytxi,sj−μˆt(rep),Lxi,sj)2;P=∑t∑i,jσˆt2xi,sj

Due to the normality inherent in the DLM, both of our moments, and hence the G and P scores, can be computed analytically without requiring samples. Hence, μt(rep)=EYt(rep)∣Y1,…,Tμˆt(rep),L=Ftht (the values of G do not depend on the covariance structure Σ), while σt2xi,sj=EYt(rep)∣Y1,…,Tσˆt2xi,sj takes the following forms for each model: σt,IW2xi,sj=Ht,iiDT,jjnT−2;σt,IG2xi,sj=Ht,iidTRjjnT−1;σt,id2xi,sj=Ht,ii; where σt,IW2,σt,IG2 and σt,id2 denote individual terms in P corresponding to Σ being inverse-Wishart, Σ=σ2R and Σ=I, respectively. Summed over the N inputs xi and S spatial coordinates sj, we write G=∑tYt−μt(rep)F2‖⋅‖F is the Frobenius norm) and P as PIW=trDTnT−2∑ttrHt;PIG=dTtr(R)nT−1∑ttrHt;Pid=S∑ttrHt

As in (22), each of the terms P. in (23) denotes the P for the model corresponding to the variance structure Σ. The lower the D-score of the distribution, the better the fit of the data to the model. Table 2 shows the results of this statistic for the FFBS applied to the data.

A Bayesian Transfer Learning Approach for BIG DATA settings

The problem of emulating high-dimensional computer model outputs has been comprehensively addressed by Higdon et al. (2008); Gramacy and Apley (2015); Gu and Berger (2016); Sauer et al. (2023a). Most of these developments have revolved around models that achieve dimension reduction using scalable derivations of Gaussian processes. Even a cursory review reveals a significant literature on statistical methods for massive spatial datasets, which is too vast to be summarized here (see, e.g., Banerjee, 2017; Heaton et al., 2019, and references therein). Within the Bayesian setting, inference proceeds from spatial processes that scale massive data sets. Examples range from reduced-rank processes or subsets of regression approaches (Quinonero-Candela and Rasmussen, 2005; Snelson and Ghahramani, 2005; Cressie and Johannesson, 2008; Banerjee et al., 2008; Wikle, 2010; Wilson and Nickisch, 2015, and references therein), multi-resolution approaches (Nychka et al., 2015; Katzfuss, 2017), and graph-based models inducing sparsity (Datta et al., 2016a; Katzfuss and Guinness, 2021; Krock et al., 2023; Dey et al., 2022; Sauer et al., 2023a; Cao et al., 2023).

We depart from the above model-based approaches and adopt a Bayesian transfer learning approach that will divide and conquer a dataset of possibly massive dimensions, construct a sequence of smaller datasets, and then apply the FFBS algorithm described in Algorithm 5 to this sequence. In spirit, this is similar to spatial meta-kriging or predictive stacking (see Guhaniyogi et al., 2017; Guhaniyogi and Banerjee, 2018; Presicce and Banerjee, 2024, and references therein). To elucidate further, let us use the analogy of a streaming show where the data at each t=1,…,T is referred to as a “season” for the show and each season consists of K subsets of the data, which we refer to as “episodes”. This scheme is depicted in Figure 2, where the data (season) at time t is partitioned into K subsets (episodes) denoted by 𝒟1t,…,𝒟Kt. Each 𝒟kt=Yk,t,Fk,t,Gk,t,Vk,t,Wk,t, where each Yk,t is r×c,Fk,t is r×p,Gk,t is p×p,Vk,t and Wk,t are positive definite covariance matrices of dimension r×r and p×p, respectively. One convenient specification emerges naturally by treating each of the K matrices Yk,t as submatrices of Yt. We let K=k1k2, where rk1=N and ck2=S where we choose r and c to ensure that we can execute Algorithm 5 on each 𝒟k,t as a sequence of datasets over KT time points.

The preceding divide and conquer method conveniently retains the conjugate distribution theory underlying Algorithm 5. The transfer learning mechanism scales the emulation of mechanistic systems when either N or S is large. Specifying Vk,t using Gaussian processes, as described in Section 2.2, allows interpolation of Yk,t(χ˜) for arbitrary units χ˜. However, inference on the complete spatial field involving S spatial locations is precluded as the Σ matrix only captures the dependence among the c columns of Yk,t in each episode. If full inference on the spatial field is desired, then we model the S columns of the full dataset as a spatial covariance matrix from which we extract the c×c matrix corresponding to each episode in the transfer learning algorithm. More specifically, we consider the model Yk,t=Fk,tΘk,t+𝓔k,t,𝓔k,t∼ind.ℳ𝒩r×cO,Vk,t,ΣΘk,t=Gk,tΘpa[k,t]+Γk,t,Γk,t∼ind.ℳ𝒩p×cO,Wk,t,Σ, where the indices {k,t} refer to the k-th episode in season t,Vk,t and Wk,t are matrices with rows and columns corresponding to the episode k within season t and pa[k,t] denotes the index that immediately precedes (is the “parent” of) the index {k,t} and defined as pak,t=k−1,tifk=2,…,K;t=1,2,…,TK,t−1ifk=1;t=2,…,T0,0ifk=1,t=1.

Algorithm 5 requires that we maintain a shared covariance matrix among the columns of Yk,t for each k and t, which is possible only if each episode has the same number of columns. Since each Yk,t is r×c, it follows that the c×c covariance matrix Σ models the covariance among the columns for every episode in the entire stream.

To analyze the computational complexity of Algorithm 5 under the transfer learning framework, we replace the number of locations with c and the number of time points with K×T. This substitution yields a computational cost of ∼OKTLc3. The primary benefit of transfer learning in this context is the reduction of the cubic term in complexity from S3 to Kc3, which can substantially alleviate the computational burden.

Applications for EmulationPredator-prey analysis

We apply our methodology to a multivariate dynamical system specified through a set of coupled ordinary differential equations. This may be considered a multivariate statistical modeling problem in lieu of spatial. The numerical solution to this dynamical system - which we call our computer model - is cheap to evaluate. To motivate the ecological computer model, we overview the Lotka-Volterra equations, which models the interaction between the populations of a single predator species and a single prey species: dutdt=η1ut−η2utvtdvtdt=−η3vt+η4utvt, where η1 is the growth rate of the prey population and η3 is the rate of population loss within the predator population independent of the interactions the two populations would have with one another; η2 and η4 represent the respective rates at which the prey population shrinks and the predator population grows, which is represented through the product of their respective populations. The parameters η1,η2,η3, and η4 must be positive; furthermore, the values of η1,…,η4 must correspond to the units of the quantities they multiply. Thus, η2 and η4, in particular, should be one or two orders of magnitude smaller than η1 and η3, since they correspond to the product of the populations of the predators and prey.

To train the FFBS model using Algorithm 5, we generate N=50 sets of lognormal sampled parameters, so that ηi∼exp𝒩μi,σi for i=1,…,4, where μi and σi are the respective means and standard deviations of the normal random variables prior to exponentiation. The training data is generated over a Latin hypercube design from the multivariate lognormal with μ1=μ3=0 and μ2=μ4=−3, and σ1=σ2=σ3=σ4=0.5, so that η1 and η3 will be close to 1 and η2 and η4, which control the prey’s population decline and the predator’s population growth respectively, will be close to 0.05.

We let Yt be the N×2 matrix, where row i corresponds to the log-transformed solutions ut,vt of (25) for the ith training parameter ηi,i=1,…,N. The design matrix for the model approximation is the data matrix from the prior time point in an AR(1) setup, so that Ft=Yt−1 for t=1,…,T (see, e.g., West and Harrison, 1997, for details of such formulations). We set Y0 to the log-population of the Canadian lynx from the data in the year 1900 and emulate Yt from (25) over T=20 time points. We also generate Vt=Cηi;ηj;β using βi=β in (13) with β=30.5×dmax, where dmax is the maximum distance between ηi ‘s, and we let Gt and Wt be identity matrices.

Also of interest in fitting Algorithm 5 to the data is a measure for the covariance between the log-predator and log-prey populations, which enables us to compute a correlation measure to interpret the results we have been given. Figure 3 presents the posterior distributions using 20,000 posterior samples of Σ. The left and right figures are the posterior samples of the diagonal elements, Σ11 and Σ22, of Σ while the middle panel is the correlation Σ12/Σ11Σ22. The 95% credible interval for the correlation is (−0.179, −0.056), which complies with the dynamics of the predator and prey populations in the Lotka-Volterra cycle. As the prey population increases, the predator population increases at the expense of the prey population, enough to cause the prey to decrease dramatically. Once the prey become scarce, the predators also begin to die off when there are less prey to eat, allowing for the prey population to grow, after which the cycle repeats.

Analytic solutions with their credible intervals are sampled and plotted in Figure 4. The bottom two plots feature the comparison of log-population values between the actual values of the log-prey and log-predator populations and their estimates produced by the FFBS emulator respectively. The 95% credible intervals accurately capture the trajectories of the predator and prey populations; particularly for the middle and right plots.

Nonlinear Partial Differential Equation Emulation

We apply our methodology to a spatiotemporal model arising from a set of coupled nonlinear PDEs, a natural way to describe the dynamics of a continuous outcome across space and time. Our model uses the set of coupled PDEs shown in (26). This mechanistic system is popular within the applied mathematics community to capture qualitative behavior of spreading infection dynamics (Noble, 1974; Keeling and Rohani, 2008). Literature studying such systems of PDEs often focuses on proving various mathematical properties with no discussion about parameter inference or connections to real-world data or field measurements. Specifically, Zhang and Wang (2014) posit the existence of traveling waves in an influenza outbreak for a system such as (26), but omit discussion of parameter inference. See Figure 6 for a visualization of this traveling wave phenomenon when two seeding locations are used as initial sources of the disease. To motivate the system of equations under study, we modify a standard SIR compartment model by including a Laplacian term to induce diffusion of the populations over space. In this model, disease transmission is predominantly a localized process where transmission is most likely between nearby locations. The movement of individuals then facilitates the geographical spread of infectious diseases. Such a process may be captured through the following system ∂St(s)∂t=−η1St(s)It(s)N+α1∇2St(s)∂It(s)∂t=η1St(s)It(s)N−η2It(s)+α2∇2It(s)∂Rt(s)∂t=η2Its+α3∇2Rts, where ∇2ft(s)=∑i=1d∂2ft(s)/∂si2. In applying our methodology, the training data is generated through the numerical solution of (26) for fixed parameter setting x=η1′,η2′,α1′,α2′,α3′. The outcome yt(s,x) denotes the infection counts at a location and time, namely It(s).

We employ deSolve (R package version 1.40) (Soetaert et al., 2010) to solve the system over the range of input parameters. Our spatial domain consists of a 100×100 grid, resulting in S=10,000 spatial locations. The system’s initial state begins with two seeding locations: one in the middle and one in the bottom left corner. We generated N=50 sets of x used in equation (26), and for each input generated solutions over T=50 time points from the PDE system, resulting in 51 time points in total including time 0.

Letting the number of infected people at time t over N inputs and S spatial coordinates be the N×S matrix It, we set the data matrix Yt=logIt+1. We further adopt a second-order autoregressive (AR2) structure for the covariate matrix Ft, but structure it so that each episode has access to data from the previous two seasons, so that Fk,t=Yk,t−1,Yk,t−2. As with the predator-prey example in the preceding section, we also specify Vk,t=Cηi,ηj;β, where β is defined the same way, as well as Gt and Wt as the identity matrices. We use 40 generated parameters to train the FFBS algorithm (Algorithm 5), taking L=10 samples in the backward sampling step, and employ a Gaussian Process to emulate the PDE solution. We use the transfer learning parameters specified in Section 4 set as r=40 and c=100, so that we regress on one column of the grid at each step to efficiently emulate our PDE.

We emulate the space-time field of the PDE using three different variance structures: (i) Σ follows inverse-Wishart; (ii) Σ=σ2R; and (iii) Σ=I. Table 1 shows the WAIC and its component statistics. The inverse-Wishart model outperforms the inverse-Gamma and identity covariance. Figure 5 shows the details of WAIC over episode/time for FFBS models with three different variance structures. Table 2 presents the GPD scores (D=G+P from (21)) obtained from independent replicates. The inverse-Wishart model performs slightly better than the inverse-Gamma model, and both outperform the identity model.

Figure 6 shows an example of the dynamics of Equation (26) generated by deSolve. The disease spreads over time to cover nearly the entire grid at t=37, but the people closest to the source points have already begun to recover. By the final time point of the simulation t=50, nearly everyone in the grid has recovered from the disease, with its prevalence being restricted to the corners of the grid, which caught the disease later than the other points.

Figure 7 compares the PDE solution and the FFBS emulation results using the ℳ𝒩ℐ𝒲 covariance structure at the same selected time points. The boundaries and overall region of the spread of the disease is accurately emulated, with some inaccuracies taking place within the infected regions. By t=31, however, even the number of infected people within the infected region has been accurately emulated, and by t=43, the emulation is nearly perfect as revealed by comparing the fields in the second row of Figure 7 with those in the first row (also corresponding to t=18,31,43 in Figure 6).

Figure 8 contrasts the PDE solution and FFBS emulated values across all locations, with the same parameters as in Figure 7. Both figures demonstrate good predictive accuracy of the FFBS algorithm in emulating the nonlinear PDE.

Estimating mechanistic system parameters using field data

The above emulator is now melded with the field observations to infer about the mechanistic parameters. Since the mechanistic system may not adequately describe observed field data over all spatial locations and time points, we include a spatiotemporally-varying discrepancy term that captures the bias between the observed field data and emulator. Our state-space model with dynamic bias correction u is zt(s)=yt(η,s)+ut(s)+εtz(s),εtz(s)∼ind.𝒩0,τt2ut(s)=ut−1(s)+εtu(s),εtu(s)∼ind.𝒢𝒫0,τt2C(⋅,⋅;ρ)τt2∼ind.ℐ𝒢btn0z,btd0z,t=1,2,…,T where yt(η,s) denotes the emulator prediction at mechanistic input η and location s. We assume that the field data are available over a set of S˜ spatial locations in a set 𝒮˜⊆𝒮. It is worth pointing out that in the traditional exercise of calibrating computer models, the discrepancy term is treated as function of the model inputs. One could certainly incorporate ut(η,s) in (27). However, such paradigms assume that the field data are partial realizations of a process with some unknown “optimal” η that is adjusted by the discrepancy terms and the noise. We do not need such assumptions for inferring about η. Instead, we treat the problem as one of a mixed nonlinear regression with the field observations regressed by the mechanistic output for some unknown η. The space-time discrepancy term now serves as a process that attempts to learn from the mechanistic system and the data.

Let ut be the S˜×1 vector with elements utsi and U(ρ) be the S˜×S˜ correlation matrix built using C(⋅,⋅;ρ). The posterior distribution conditional on the collection of field observations, z1:T, and the emulated values, Y1:T, is proportional to the joint distribution p(η,ρ)×𝒩𝒢u0,τ0−2∣m0z,M0z,n0z,d0z×∏t=1T𝒩𝒢ut,τt−2∣ut−1,U(ρ),btn0z,btd0z×∏t=1T𝒩yt(η)∣μ˜t(η),Σ˜(η)∏i=1S˜𝒩ztsi∣ytη,si+utsi,τt2, where yt(η) is the S˜×1 vector with elements yt(η,s) for each location s∈𝒮˜,μ˜t(η)⊤=F˜t(η)Θt(𝒮˜)+Jt(η)⊤Vt−1Yt(𝒮˜)−FtΘt(𝒮˜) and Σ˜t(η)=(1−Jt(η)⊤Vt−1Jt(η))Σ(𝒮˜) are the 1×S˜ mean vector and S˜×S˜ covariance matrix, respectively, derived from the conditional predictive distribution in (11) for yt(η) given the emulated values Y1:T,F˜(η) is 1×p with entries equal to the row in F˙t corresponding to η,Θt(𝒮˜) are Yt(𝒮˜) are p×S˜ and N×𝒮˜ consisting of columns corresponding to locations in 𝒮˜ extracted from Θt and Yt, respectively, Jt(η) is N×1 with jth element given by the covariance function Cxi,η;β in (13) and Vt is N×N as defined in Section 2.2. The definition of Σ(𝒮˜) depends upon our specific choice of the emulation model. For example, if we use (2) for emulation and if the field data locations in 𝒮˜ are a subset of the S locations in 𝒮 that were used for emulation, then Σ(𝒮˜) is the S˜×S˜ sub-matrix of Σ corresponding to the locations in 𝒮˜. Alternatively, if one uses (14) for emulation, then Σ(𝒮˜)=σ2R(𝒮˜), where R(𝒮˜) is the S˜×S˜ spatial correlation matrix formed over the locations in 𝒮˜ using a spatial correlation function; the field data locations need not be a subset of emulator locations.

Full conditional distributions for pτ1:T2∣⋅.

1:function TauSq_postu0:T,y1:T,z1:T,n0,d0,b,U
2: for t=1 to T do
3:  Q1←ut−ut−1⊤U−1ut−ut−1,Q2←zt−yt−ut⊤zt−yt−ut▷ O(𝒮˜2)
4:  Sample τt2∼ℐ𝒢btn0+S˜,btd0+12Q1+Q2
5: end for
6: return τ1:T2
7:end function▷ O(T𝒮˜2)

We draw samples from pη,ρ,τ0:T2,y1:T(η),u0:T∣z1:T,Y1:T given by ∫pη,ρ,τ0:T2,u0:T,y1:T(η),Σ,Θ0:T∣z1:T,Y1:TdΣdΘ0:T=∫pη,ρ,τ0:T2,u0:T,y1:T(η)∣Σ,Θ0:T,z1:T,Y1:T×pΣ,Θ0:T∣z1:T,Y1:T⎵Modularize:pΣ,Θ0:T∣Y1:TdΣdΘ0:T where y1:T(η)=(y1(η)⊤,…,yT(η)⊤)⊤ is the S˜T×1 vector with yt(η) as the S˜×1 vector with elements ytη,si for i=1,…,S˜. A pragmatic approach to sampling from (29) relies on the posterior in (28) and the notion of Bayesian Modularization (Bayarri et al., 2009). Modularization replaces the underlined expression in (29) with the lower dimensional distribution pΣ,Θ0:T∣Y1:T, which is readily sampled through FFBS and Metropolis-Hastings steps of Algorithm 11. For each drawn value of {Σ,Θ}, we will need to draw from pη,ρ,τ0:T2,u0:T,y1:T(η)∣Σ,Θ0:T,z1:T,Y1:T.

Drawing samples from pη,ρ,τ0:T2,u0:T,y1:T(η)∣Σ,Θ0:T,z1:T,Y1:T employs Gibbs updates using full conditional distributions for y1:T(η) and τ0:T2, FFBS updates for u0:T, and Metropolis random-walk updates for η and ρ.

The full conditional distributions for τt2∣⋅ are ℐ𝒢ntz,dtz where ntz=S˜+btn0z and dtz=btd0z+12{ut−ut−1⊤U(ρ)−1ut−ut−1+∑i=1S˜ztsi−ytη,si−usi2} for t=1,…,T and i=1,…,S˜. The full conditional distributions for yt(η)∣⋅ are 𝒩Bt(η)bt(η),Bt(η) where Bt(η)−1=Σ˜t(η)−1+τt−2IS˜ and bt(η)=Σ˜t(η)−1μ˜t(η)+τt−2zt−ut with μ˜t(η)=μ˜tη,s1,…,μ˜tη,sS˜⊤. Algorithm 6 provides the steps to compute and draw samples from the full conditional distributions of τ1:T2∣⋅ with a computational cost of O(TS˜2). Algorithm 7 provides the steps to compute and draw samples from the full conditionals for y1:T∣· with a computational cost of O(TS˜3).

Full conditional distributions for py1:T∣⋅.

1:function Y_POSTη,τ1:T2,z1:T,u1:T,𝒮˜
2: for t=1 to T do
3:  μ˜t(η)⊤←F˜t(η)Θt(𝒮˜)+Jt(η)⊤Vt−1Yt(𝒮˜)−FtΘt(𝒮˜)▷ O(pN𝒮˜)
4:  Σ˜t(η)←1−Jt(η)⊤Vt−1Jt(η)Σ(𝒮˜)▷ ON2
5:  Bt(η)←(Σ˜t(η)−1+τt−2IS˜)−1,bt(η)←Σ˜t(η)−1μ˜t(η)+τt−2zt−ut▷ O(𝒮˜3)
6:  Sample yt∼𝒩Btbt,Bt▷ O(𝒮˜3)
7: end for
8: return y1:T,B1:T,b1:T
9:end function▷ O(T𝒮˜3)

The model bias process realizations, u0:T, are updated from its joint full conditional, pu0:T∣⋅ using the FFBS algorithm. As with emulation, we compute the moments for ut∣⋅,z1:t,y1:t(η) using the Forward Filter, and then acquire the smoothed moments of ut∣⋅,z1:T,y1:T(η) using Backwards Sampling. Note that ut are vectors relevant to Gibbs sampling, we only need one sample of u0:T∣⋅ per iteration. Algorithm 8 outlines the steps for generating samples of u0:T∣⋅,z1:T,y1:T(η) using FFBS algorithm. Lines 5–12 describe the Forward Filtering steps, with a computational cost of ∼O(TS˜2). Similarly, Lines 14–21 detail the Backward Sampling steps, also with a cost of ∼OTS˜2). Thus, the total computational cost is ∼O(TS˜2).

Finally, we consider the density for ρ,η∣⋅, which we glean from the relevant terms in (28). Unfortunately, even when we choose comparatively simple priors for p(ρ,η) (e.g. multivariate uniform distributions), there is no recognized density that would enable us to sample from known distributions. Consequently, we update ρ and η at each iteration via the Metropolis algorithm. Algorithm 10 gives steps for the Metropolis random walk. The target density is given in Algorithm 9, which corresponds to terms in (28).

Since our use cases depend on the entries of ρ and η being positive, or at the very least nonnegative, we first transform the parameters to the real line and use a normal density for proposing updates in the Metropolis sampler. Let ϕ=ρ⊤,η⊤⊤ be the concatenation of ρ and η,lϕ be the total number of elements of ρ and η,δ←𝒩0,ϵ32I denote the proposal update, and g(⋅) be the function rescaling each element of ϕ to the real line. The resulting transformation necessitates the jacobian 𝒥ϕ(g(ϕ)) for both ϕ and ϕ*←g−1(g(ϕ)+δ), with g(ϕ) and g(ϕ*) considered distributed on 𝒩(0,1).

We can employ Bayesian modularization to prevent the model discrepancy term ut from interfering with inference on mechanistic parameters. This additional modularization treats the emulator as an unbiased representation of the field data. After fully sampling the model parameters, each update of the dynamic bias term conditions on a draw from the posterior of the parameters to model the discrepancy between the field data and emulator predictive distribution. Algorithm 11 outlines the steps for learning about the mechanistic system parameters using field data. Each iteration updates the parameters τ1:T2,u1:T,ρ and η (via ϕ), and y1:T using the updated values in order; Line 7 in particular proposes new values using a Metropolis update. Repeating the process for L iterations results in a total computational cost of O(LT𝒮˜3).

FFBS for calibration with computer model bias

1:Input: Field data z1:T, emulation results y1:T(η), starting values m0z,τ02,M0z, data scale variance τ1:T2, and correlation matrix U.
2:Output: Calibration samples u0:T
3:function FFBS_calibrationz1:T,y1:T(η),m0z,τ02M0z,τ1:T2,U
4: # Forward Filter
5: for t=1 to T do
6:  # Compute prior distribution covariance matrix
7:  A˜t←τt−12Mt−1z+τt2U▷ O(S˜2)
8:  # Compute one-step ahead forecast covariance matrix
9:  Q˜t←A˜˜t+τt2IS˜▷ O(S˜)
10:  # Compute filtering distribution moments
11:  mtz←mt−1z+A˜tQ˜t−1(zt−yt(η)−mt−1z);Mtz←τt−2(A˜t−A˜tQ˜t−1A˜t)▷ O(S˜3)
12: end for
13: # Backwards Sampling
14: h˜T←mTz;H˜T←τT2MTz
15: Sample uT∼𝒩h˜T,H˜T▷ O(S˜3)
16: for t=T−1 to 0 do
17:  # Compute BS smoothing distribution moments
18:  h˜t←mtz+τt2MtzA˜t+1−1h˜t+1−mtz;▷ O(S˜3)
19:  H˜t←τt2Mtz−τt4MtzA˜t+1−1A˜t+1−H˜t+1A˜t+1−1Mtz▷ O(S˜3)
20:  Sample ut∼𝒩h˜t,H˜t▷ O(S˜3)
21: end for
22: return u0:T
23:end function▷ O(TS˜3)

Log probability density of the full conditional p(ϕ|·).

1:function LOGLIKϕ:=(ρ,η),τ1:T2,u1:T,z1:T,𝒮˜
2: y1:T,B1:T,b1:T←y_postη,τ1:T2,z1:T,u1:T,𝒮˜▷ Algorithm 7
3: 𝒥(ϕ)←det∂g(ϕ)∂ϕ
4: logp(ϕ∣⋅)=∑t=1Tlog𝒩yt∣Btbt,Bt+∑i=1S˜log𝒩ztsi∣ytsi+utsi,τt2+log𝒩ut∣ut−1,τt2U(ρ)+logp(ϕ)−log𝒥(ϕ)
5:
6: return logp(ϕ∣⋅)
7:end function▷ O(T𝒮˜3)

Applications for calibrationPredator-prey analysis

We estimate parameters in the Lotka-Volterra equations in (25). We fix ρ=1.5 and, analogous to Section 5.1, set the prior for η to be a multivariate lognormal. Then giηi=logηi−μi/σi is the function that takes in η and outputs a 4-dimensional vector into R4 that is distributed 𝒩4(0,I), and its Jacobian takes the simple form 𝒥(η)=∏i=14ηiσi−1.

Metropolis random walk algorithm (one step).

1:function Metrop(θ,p(),Υ)
2: Draw θ*∼q(⋅∣θ):=N(⋅∣θ,Υ);
3: Set θ=θ* with probability α=min{1,exp[logp(θ*∣⋅)−logp(θ∣⋅)]}.
4: return θ
5:end function

Sampler for estimating mechanistic system parameters using field data

1:Given: Field data z1:T, emulation data Y1:T,L posterior samples of pΘ0:T,Σ∣Y1:T from Algorithm 5, functions F˜1:T,J1:T,g, hyperparameters n0,d0,b,τ02,u0,ρ(0),η(0),m0z,M0z,U,Υ.
2:Initialize: ϕ(0)=gρ(0),η(0),τ1:T2,(0),u1:T(0),y1:T(0).
3:for l=1 to L do
4: τ1:T2,(l)←TauSq_postu0:T(l−1),y1:T(l−1),z1:T,n0,d0,b,U▷ Algorithm 6
5: u1:T(l)←FFBS_calibrationτ0:T2,(l),y1:T(l−1),z1:T,m0z,M0z,U▷ Algorithm 8
6: ϕ(l)←Metropϕ(l−1),loglik,Υ▷ Algorithm 9,10
7: y1:T(l)←y_postη(l),τ1:T2,(l),u1:T(l),z1:T,𝒮˜▷ Algorithm 7
8:end for▷ O(LT𝒮˜3)

For our noisy field observations, we utilize the data recorded on the Canadian lynx and snowshoe hare population sizes. (Hewitt, 1917) We take our training data and emulation parameters from Section 5.1 for calibration, with N=50 lognormal η sampled with a latin square design and T=20. Figure 9 plots the simulated prey and predator log-populations with the ground-truth log-populations of the hares and lynxes respectively. The agreement between the collection of curves and the real data is remarkable especially given that only the initial population was used from the real data to generate the curves and the Lotka-Volterra parameters were independently generated.

Figure 10 displays calibration samples on real lynx and hare population data collected over a 20-year period. The samples are generated using Algorithm 11 with L=20,000. The resulting medians generated from the quantiles are: 0.308 (0.036, 1.224), 0.054 (0.008, 0.340), 0.305 (0.09, 2.189), and 0.039 (0.012, 0.097). Taking the medians as the point estimates for the calibrated solutions of the real-life lynx and hare population data, the number of hares grows at a rate of 0.308 times its population that year every year independent of the presence of any predators, but decreases at the rate of 0.054 times the populations of the hares and lynxes per year due to predation from the lynxes. At the same time, the number of lynxes decreases at the rate of 0.305 times its population that year, but grows at the rate of 0.039 times the populations of the hares and lynxes that year, due to eating the hares.

Nonlinear Partial Differential Equation Calibration

Applying calibration to the SIR model equation (26), we fix ρ to be constant (approximately 0.85) and set the prior of η to be a multivariate uniform distribution ∏i=15Uai,bi, where ai and bi are the limits of the uniform distribution for each parameter. Then giηi=logitηi−ai/bi−ai) and the Jacobian takes on the form: 𝒥(ϕ)=∏i=1lϕηi−ai−1+bi−ηi−1.

In this example, we consider a 6×6 grid with 36 locations and 26 time points, including time 0, and 50 inputs η from a deterministic system. Carrying over our analysis from emulation in Section 5, we adopt the same AR2 structure for Ft. We use 30,000 samples, with the transfer learning parameters specified in Section 4 set as r=50 and c=2. In calibration, we set α1=α3=0 to simplify the analysis.

Figure 11 shows the histograms of posterior densities for elements of parameters η indicating that 95% credible intervals cover the true value and shrink our belief of distribution of η to a narrower interval. The posterior medians (95% credible intervals) of ‘s are 2.91 (2.40, 3.35) for η1,0.30(0.24, 0.37) for η2, and 0.11 (0.07, 0.20) for α2, respectively. Figure 12 compares the emulation field generated by true and 95% credible intervals of η. The 2.5% percentile of posterior values of η shows a slower process in infection and recovery, while 97.5% percentile shows a faster rate in infection and recovery over space. Figure 13 shows the posterior variability, spatial bias, and field predictions in calibration models. The discrepancy in τ2 with the true value arises from using only a single realization. However, both the spatial bias and field values show good alignment with the true data.

Calibrating a Computer Simulation with Network Output

Network science studies how the macro-structure of interconnected systems such as telecommunication, economic, cognitive, or social networks affect global dynamics (Watts and Strogatz, 1998). Within a network, nodes represent discrete entities, and edges represent relations. Of prominent interest in applied modeling is understanding how global structure affects spread of a quantity throughout the system, termed network activation or network diffusion. To date, there exist various statistical software packages implementing diffusion or contagion processes across networks, but less attention has been given to calibrating these models to real-world data (Vega Yon and Valente, 2021; Siew, 2019). In this example, we focus on calibrating a deterministic computer model popular within psychology and psycholinguistics communities (Chan and Vitevitch, 2009; Vitevitch et al., 2011). The spreadr package (Siew, 2019) implements a network diffusion simulation but ignores the calibration problem, justifying use of our methodology.

We first define notation necessary for understanding the inputs to this computer simulation. For a network with n nodes, two parameters control the activation spread, known as retention and decay, denoted as r and d respectively.

r - a scalar or a vector of length n that controls the proportion of activation retained by a node at each time step of the simulation.

d - a scalar that controls the proportion of activation that is lost at each time step of the simulation.

There are three quantities of interest that define how activation spreads dynamically over time as functions of r and d: reservoir, outflow, and inflow. These are defined as follows:

reservoir(t,n)=r×inflow(t,n).

outflowt,n=1−d1−r×inflowt,ndegn.

inflow (t,n)=∑i=1deg(n) outflow t−1,ni+reservoir(t−1,n), where deg(n) denotes the number of connections to node n.

The computer simulation then computes these quantities for each node in the network across time. A visualization of the spatiotemporal spreading dynamics is show in Figure 14.

We employ a space-filling design to generate a small collection of plausible parameter values. Subsequently, we select a random validation point that is not included in the training set. After 10,000 posterior draws, the prior-to-posterior learning is evident in Figure 15 and demonstrates our methodology is capable of calibrating a computer model generating dynamics across a network. Although it is possible to define a Gaussian process over a graph (Venkitaraman et al., 2020), for this application we fix the correlation structure to arise from the adjacency matrix. Our results demonstrate that failing to account for spatial structure leads to information loss. Specifically, we observe a 13.7% improvement in RMSE, decreasing from 0.110 to 0.095, when using the spatial model compared to the heterogeneous model, based on 25 computer model runs.

Concluding remarks

This manuscript demonstrates how statistical distributions can be used to devise a viable and effective probabilistic learning framework for emulating dynamically evolving spatiotemporal mechanistic systems without resorting to iterative inferential algorithms. Subsequently, we offer modularized calibration from field observations by regressing them on the emulated field. While this step requires MCMC for full probabilistic inference on the mechanistic model parameters, we do not need to recompute or update the emulated field. Building upon a rich literature on state-space extensions to nonlinear systems (such as nonlinear PDEs explored in Wang et al., 2021, but without melding of field observations), our approach can learn about mechanistic model parameters using more accurate and flexible emulators. Since we do not require accessibility to the mechanistic system (only its output is needed), this framework is equally applicable to systems built on mathematical models from physical principles and to agent-based models driven by computer programs (e.g., our network example).

A key point of our framework is that we emulate the mechanistic system using analytically accessible matrix-variate statistical distributions without resorting to iterative algorithms that may consume resources until they converge. Furthermore, scaling to large number of locations and time points is also achieved in closed form using Bayesian transfer learning with the FFBS algorithm. Full uncertainty quantification for learning the mechanistic system’s parameters will involve MCMC, but modularization again helps with computational feasibility. In this regard, our framework holds promise as an effective trainer for generative Artificially Intelligent engines such as “BayesFlow” (Radev et al., 2022) or neural Bayes (Sainsbury-Dale et al., 2024; Zammit-Mangion et al., 2024). Such developments pertaining to the use of our framework for amortized inference will constitute future research.

In the interest of retaining exact inference, we fixed the covariance kernel hyper-parameters in the Gaussian processes used to emulate the system. This does not affect the emulated fields since, as we have remarked, the Gaussian process interpolates the field irrespective of the values of the hyper-parameters. Fixing these parameters can be achieved using exploratory methods for stochastic processes such as variograms (Zhang et al., 2019). Alternatively, we can use predictive stacking to arrive at posterior distributions averaged over fixed values of the hyper-parameters (Zhang et al., 2024; Presicce and Banerjee, 2024). These approaches will still offer emulation without resorting to iterative MCMC algorithms. Alternatively, sequential screening procedure may be employed to identify the most influential kernel parameters (Welch et al., 1992; Merrill et al., 2021).

Learning from massive amounts of high-resolution spatial-temporal field data has not been explicitly addressed here, but the rapidly evolving literature on scaling Gaussian processes to massive data is relevant. While traditional low-rank or inducing point methods such as Snelson and Ghahramani (2005), Wilson and Nickisch (2015) or the Gaussian predictive process (Banerjee et al., 2008) remain feasible either over the set of spatial locations or over the space of inputs, it is well established that such methods can over-smooth when the number of outputs are very large (Banerjee, 2017) and multi-resolution adaptations (Nychka et al., 2015; Katzfuss, 2017) are needed to mitigate over-smoothing. Sparsity-inducing processes such as the Nearest-Neighbor Gaussian processes (Datta et al., 2016a,b; Finley et al., 2019; Peruzzi et al., 2022) or the similarly themed Vecchia-based processes (Katzfuss and Guinness, 2021; Sauer et al., 2023b,a; Cao et al., 2023) offer greater scalability and can be used to model ut(⋅) in Section 6. Alternatively, we can extend the divide and conquer paradigm used for emulation (Section 4) to calibration. However, the transfer learning framework may be hampered by irregular space-time coordinates. Future directions will explore spatial meta-kriging (Guhaniyogi and Banerjee, 2018; Guhaniyogi et al., 2017) and predictive stacking (Presicce and Banerjee, 2024; Pan et al., 2024).

Other extensions include treating outputs on different coordinates when adaptive grids are used to solve PDEs yielding misaligned outputs, i.e., not all runs are made over the same set of locations. Here, one may need to vectorize the incomplete output matrices and use multivariate normal distributions that would render learning using process-based misalignment models (see, e.g., Chapter 7 in Banerjee et al., 2014). High-dimensional spatial outputs can be treated using dimension-reduction methods such as spatial factor models (Ren and Banerjee, 2013; Zhang and Banerjee, 2022). One could also generalize the model to have different variances at different spatial grids. In fact, richer inference is possible using MCMC algorithms that we have avoided (except to execute modularized calibration for the mechanistic parameters). The RobustGaSP package developed in Gu et al. (2019) is a viable candidate to achieve effective inference in these settings. Extensions to generalized dynamic linear models (West et al., 1985; Gamerman et al., 2013) will include analyzing non-Gaussian data and investigating richer specifications for Gt and Wt. We can adapt our approach using vectorized multivariate Gaussian distributions to emulate mechanistic systems at unseen spatial locations and estimate (calibrate) in spatially misaligned settings, where the emulation locations and field measurements do not match (Banerjee et al., 2014). Lastly, the burgeoning field of sequential Monte Carlo analysis is applicable to build more general statistical emulators within this framework (Hirt and Dellaportas, 2019).

Computing environments and timings

We developed our methods in C++ with functions available in R (version 4.4.1) employing the Rcpp package (Eddelbuettel and François, 2011). Computations for the predator-prey system were executed on a laptop running 64-bit Windows 11 with a 12th Gen Intel(R) Core(TM) i7–12700H, 2.30 GHz processor, 32.0 GB RAM at 4800 MHz, 6 GB Graphics Card equipped with multiple GPUs. Computations pertaining to the PDE and network diffusion examples were executed on a laptop running macOS 14.7.6, equipped with an Apple M3 Max processor (16-core CPU, 40-core GPU) and 64 GB of RAM. All computer programs required to reproduce the numerical results in this manuscript are available from the GitHub repository https://github.com/xiangchen-stat/Bayesian_Modeling_Mechanistic_Systems.

Empirical computation time required to solve and emulate the Lotka-Volterra ODE system, as described in Section 5.1, was approximately 0.26 and 2.16 seconds, respectively. Computations described in Section 5.2 for the PDE system required approximately 9 minutes and 13 seconds for solving the system, and 21, 61, and 59 seconds to emulate the field for the three considered variance structures, respectively. Calibration experiments, as described in Section 7, delivered full posterior inference in 98 minutes for the predator-prey system, 59 minutes for the PDE system, and about 14 and 25 seconds for the non-spatial and spatial models in the network diffusion system, respectively.

Acknowledgments

Sudipto Banerjee acknowledges funding from the National Science Foundation through DMS-1916349 and DMS-2113778; and funding from the National Institute of Health through NIEHS-R01ES027027, NIEHS-R01ES030210 and NIGMS-R01GM148761. The work of all the authors were supported, in part, from these funds.

In the rest of the article, we occasionally refer to the mechanistic system which we seek to model, learn, and estimate parameters from field data as a computer model to be consistent with terminology in statistics for referring to deterministic models.

AppendixDistributions in the FFBS <xref rid="T5" ref-type="table">Algorithms 3</xref>, <xref rid="T6" ref-type="table">4</xref> and <xref rid="T7" ref-type="table">5</xref>

The joint posterior distribution (4) is obtained by exploiting familiar distribution theory from statistical linear models. The first two equations in (2) implies a well-defined joint distribution for Yt and Θt conditional on Y1:(t−1) vecYtvecΘtY1:(t−1),Σ∼𝒩N+pS×1vecqtvecat,Σ⊗QtΣ⊗FtAtΣ⊗AtFt⊤Σ⊗At.

Using familiar properties of the multivariate normal distribution, we obtain vecΘt∣Y1:t,Σ∼𝒩pS×1vecat+IS⊗AtFt⊤Qt−1vecYt−qt,Σ⊗Mt, which implies Θt∣Yt,Σ∼ℳ𝒩mt,Mt,Σ, where mt=at+AtFt⊤Qt−1Yt−qt and Mt=At−AtFt⊤Qt−1FtAt. The posterior distribution of Σ∣Y1:t is obtained from pΣ∣Y1:t=ℐ𝒲Σ∣nt−1,Dt−1⏟pΣ∣Y1:(t−1)×ℳ𝒩Yt∣qt,Qt,Σ⏟pYt∣Y1:(t−1),Σ∝exp−12trDt−1Σ−1(det(Σ))nt−1+S+12×exp−12trYt−qt⊤Qt−1Yt−qtΣ−1(det(Σ))N2∝exp−12trDtΣ−1det(Σ)nt+S+12∝ℐ𝒲Σ∣nt,Dt, where nt=nt−1+N and Dt=Dt−1+Yt−qt⊤Qt−1Yt−qt. Forward sampling computes the above quantities recursively to arrive at (31) and (32) for t=T and draws L samples of Σ from (32) followed by one draw of ΘT from (31) for each of the L values of Σ.

The forward sampling described above yields L samples of ΘT,Σ from (4). For each of the L values, we will repeat an entire cycle of backward sampling. From (7), backward sampling will draw from pΘt∣Σ,Θt+1,Y1:t, which can be achieved by drawing one value of Θt for each of the L sampled values of Θt+1,Σ sequentially from t=T−1,…,0. Alternatively, we can draw the samples from pΘt∣Σ,Y1:t directly as described in (16). To derive this distribution, we make use of the Markovian structure in the model to note that pΘt∣Θt+1:T,Y1:T,Σ∝pΘt∣Y1:t,Σ×pΘt+1∣Θt,Σ∝pΘt∣Θt+1,Y1:t,Σ, where we have used the facts that Θt is conditionally independent of Yt+1:T given Y1:t and that Θt+1 is conditionally independent of Y1:T given Θt. The joint distribution of Θt and Θt+1 conditional on Y1:t is given by vecΘt+1vecΘtY1:t,Σ∼𝒩2pS×1vecat+1vecmt,Σ⊗At+1Σ⊗Gt+1MtΣ⊗MtGt+1⊤Σ⊗Mt.

Therefore, the conditional distribution of Θt given Θt+1,Y1:t,Σ is vecΘt∣Θt+1,Y1:t,Σ∼𝒩pS×1vecmt+IS⊗MtGt+1⊤At+1−1vecΘt+1−at+1,Σ⊗Mt−MtGt+1⊤At+1−1Gt+1Mt, which implies that pΘt∣Θt+1Y1:t,Σ is equal to ℳ𝒩Θt∣mt+MtGt+1⊤At+1−1Θt+1−at+1,Mt−MtGt+1⊤At+1−1Gt+1Mt,Σ.

This also equals pΘt∣Θt+1,Y1:T,Σ because Θt and Yt+1:T are conditionally independent given Θt+1. Denoting Θt+1∣Y1:T,Σ∼ℳ𝒩ht+1,Ht+1,Σ, where hT=mT and HT=MT, we derive the joint distribution pΘt,Θt+1∣Y1:T,Σ=pΘt+1∣Y1:T,Σ×pΘt∣Θt+1Y1:T,Σ, which is a composition of two matrix-variate linear models with independent error matrices constructed over the same probability space (conditional on Y1:T and Σ), Θt=mt+MtGt+1⊤At+1−1Θt+1−at+1+EtandΘt+1=ht+1+E˜t+1, where Et∼ℳ𝒩O,Mt−MtGt+1⊤At+1−1Gt+1Mt,Σ is distributed independently of E˜t+1∼ℳ𝒩O,Ht+1,Σ. Substituting the model for Θt+1 into the first equation in (34) followed by familiar multivariate normal calculations yields pΘt∣Y1:T,Σ=ℳ𝒩ht,Ht,Σ, where ht and Ht are defined recursively as in (16). This completes the required closed-form derivations for all the distributions in the exact FFBS algorithm.

Calculus-free derivation of <inline-formula><mml:math id="M852" display="inline"><mml:mi>ℋ</mml:mi><mml:mi>𝒯</mml:mi></mml:math></inline-formula> density

Recall that in equations (5) and (6) we were able to compute their distributions by means of marginalizing, that is, integrating, out the undesired variance terms. We demonstrate an approach to this problem without using integration. Begin with the identity p(A∣B)p(B)=p(A,B) where A and B denote random variables. Let p(A∣B)=pΣ∣Θt,Y1:T,p(B)=pΘt∣Y1:T, and p(A,B)=pΘt,Σ∣Y1:T. Dividing both sides of (35) by pΣ∣Θt,Y1:T, we get: pΘt∣Y1:T=pΘt,Σ∣Y1:TpΣ∣Θt,Y1:T

Recall that pΘt,Σ∣Y1:T is matrix-normal-inverse-Wishart with density (follows from (3):) pΘt,ΣT∣Y1:T=detDTnT/2exp−12trDT+Θt−ht⊤Ht−1Θt−htΣ−12nTS/2(2π)pS/2ΓSnT2detHtS/2(det(Σ))nT+S+p+12,

We are also given that pΣ∣Θt,Y1:T is an inverse-Wishart. More specifically, p(Σ∣Θt,Y1:T∝pΘt,Σ∣Y1:T, which means they share product terms in their respective expressions that contain Σ and Θt (and Y1:T). Hence: pΣ∣Θt,Y1:T∝exp−12trDT+Θt−ht⊤Ht−1Θt−htΣ−1(det(Σ))nT+S+p+12

Introducing the constant (non-Σ) multiplier for (38), given it is Inverse-Wishart, yields: pΣ∣Θt,Y1:T=exp−12trDT+Θt−ht⊤Ht−1Θt−htΣ−12nT+pS/2ΓSnT+p2(det(Σ))nT+S+p+12×detDT+Θt−ht⊤Ht−1Θt−htnT+p/2

Finally, we take the quotient to obtain the exact form of pΘt∣Y1:T: pΘt∣Y1:T=detDTnT/2exp−12trDT+Θt−ht⊤Ht−1Θt−htΣ−12nTS/2(2π)pS/2ΓSnT2detHtS/2(det(Σ))nT+S+p+12×2nT+pS/2ΓSnT+p2(det(Σ))nT+S+p+12exp−12trDT+Θt−ht⊤Ht−1Θt−htΣ−1×detDT+Θt−ht⊤Ht−1Θt−ht−nT+p/2=π−pS/2ΓSnT+p2ΓSnT2detDT−p/2detHtS/2×detIS+DT−1Θt−ht⊤Ht−1Θt−ht−nT+p/2

This is the Hyper-T density in (5), specifically ℋ𝒯Θt∣ht,Ht,nT,DT. We have thus demonstrated a derivation of the Hyper-T density without using integration.

Distributions in the FFBS algorithm for the model in <xref rid="FD16" ref-type="disp-formula">(14)</xref>

The joint posterior distribution corresponding to the model in (14) is obtained by modifying (30). The first two equations in (14) implies the joint distribution for Yt and Θt conditional on Y1:(t−1) as vecYtvecΘtY1:(t−1),σ2∼𝒩N+pS×1vecqtvecat,σ2R⊗QtR⊗FtAtR⊗AtFt⊤R⊗At.

We obtain the analog of (31) vecΘt∣Y1:t,σ2∼𝒩pS×1vecat+IS⊗AtFt⊤Qt−1vecYt−qt,σ2R⊗Mt, which implies Θt∣Yt,σ2∼ℳ𝒩mt,Mt,σ2R, where mt=at+AtFt⊤Qt−1Yt−qt and Mt=At−AtFt⊤Qt−1FtAt. The posterior distribution of σ2∣Y1:t is obtained as pσ2∣Y1:t=ℐ𝒢σ2∣nt−1,dt−1⏟pσ2∣Y1:(t−1)×ℳ𝒩Yt∣qt,Qt,σ2R⏟pYt∣Y1:(t−1),σ2∝1σ2nt−1+1×exp−dt−1σ2×exp−12σ2trYt−qt⊤Qt−1Yt−qtR−1σ2NS2∝1σ2(nt−1+NS2+1)×exp−1σ2dt−1+12trYt−qt⊤Qt−1Yt−qtR−1∝ℐ𝒢σ2∣nt,dt, where nt=nt−1+NS2 and dt=dt−1+12trYt−qt⊤Qt−1Yt−qtR−1. Forward sampling computes the above quantities recursively to arrive at (42) and (43) for t=T and draws L samples of σ2 from (43) followed by one draw of ΘT from (42) for each of the L values of σ2.

ReferencesAbdallaN, BanerjeeS, RamachandranG, and ArnoldS. Bayesian state space modeling of physical processes in industrial hygiene. Technometrics, 62(2):147–160, 2020. doi: 10.1080/00401706.2019.1630009. URL https://doi.org/10.1080/00401706.2019.1630009.32499665 BanerjeeS. High-dimensional Bayesian geostatistics. Bayesian analysis, 12(2):583, 2017.29391920 BanerjeeS and RoyA. Linear algebra and matrix analysis for statistics. CRC Press, Boca Raton, FL, 2014.BanerjeeS, GelfandAE, FinleyAO, and SangH. Gaussian predictive process models for large spatial data sets. Journal of the Royal Statistical Society Series B: Statistical Methodology, 70(4):825–848, 2008.19750209 BanerjeeS, CarlinB, and GelfandA. Hierarchical Modeling and Analysis for Spatial Data. CRC Press, 2014.BayarriMJ, BergerJO, CafeoJ, Garcia-DonatoG, LiuF, PalomoJ, ParthasarathyRJ, PauloR, SacksJ, and WalshD. Computer model validation with functional output. The Annals of Statistics, 35(5):1874–1906, 2007a.BayarriMJ, BergerJO, PauloR, SacksJ, CafeoJA, CavendishJ, LinC-H, and TuJ. A framework for validation of computer models. Technometrics, 49(2):138–154, 2007b.BayarriMJ, BergerJO, and LiuF. Modularization in Bayesian analysis, with emphasis on analysis of computer models. Bayesian Analysis, 4(1):119 – 150, 2009.BleiDM, KucukelbirA, and McAuliffeJD. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859–877, 2017. doi: 10.1080/01621459.2017.1285773. URL https://doi.org/10.1080/01621459.2017.1285773.CaoJ, KangM, JimenezF, SangH, SchaeferFT, and KatzfussM. Variational sparse inverse cholesky approximation for latent Gaussian processes via double kullback-leibler minimization. In KrauseA, BrunskillE, ChoK, EngelhardtB, SabatoS, and ScarlettJ, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 3559–3576. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/cao23b.html.CarterC and KohnR. On Gibbs Sampling for State Space Models. Biometrika, 1994.ChanKY and VitevitchMS. The influence of the phonological neighborhood clustering coefficient on spoken word recognition. Journal of Experimental Psychology: Human Perception and Performance, 35(6):1934, 2009.19968444 Chan-GolstonAM, BanerjeeS, and HandcockMS. Bayesian inference for finite populations under spatial process settings. Environmetrics, 31, 2020.ContiS and O’HaganA. Bayesian emulation of complex multi-output and dynamic computer models. Journal of Statistical Planning and Inference, 140(3):640–651, 2010.CressieN and JohannessonG. Fixed rank kriging for very large spatial data sets. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(1):209–226, 2008. doi: 10.1111/j.1467-9868.2007.00633.x. URL https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-9868.2007.00633.x.DattaA, BanerjeeS, FinleyAO, and GelfandAE. Hierarchical nearest-neighbor Gaussian process models for large geostatistical datasets. Journal of the American Statistical Association, 111(514):800–812, 2016a. doi: 10.1080/01621459.2015.1044091. URL https://doi.org/10.1080/01621459.2015.1044091.29720777 DattaA, BanerjeeS, FinleyAO, HammNAS, and SchaapM. Nonseparable dynamic nearest neighbor Gaussian process models for large spatio-temporal data with an application to particulate matter analysis. The Annals of Applied Statistics, 10(3):1286 – 1316, 2016b.29657659 DeyD, DattaA, and BanerjeeS. Graphical Gaussian process models for highly multivariate spatial data. Biometrika, 109(4):993–1014, 2022. ISSN 1464–3510. doi: 10.1093/biomet/asab061. URL https://doi.org/10.1093/biomet/asab061.36643962 EddelbuettelD and FrançoisR. Rcpp: Seamless R and C++ integration. Journal of Statistical Software, 2011.EleftheriadisS, NicholsonT, DeisenrothM, and HensmanJ. Identification of Gaussian process state space models. Advances in neural information processing systems, 30, 2017.FangK-T, LiR, and SudjiantoA. Design and Modeling for Computer Experiments. Chapman and Hall/CRC, oct 2005.FarahM, BirrellP, ContiS, and AngelisDD. Bayesian emulation and calibration of a dynamic epidemic model for a/h1n1 influenza. Journal of the American Statistical Association, 109(508):1398–1411, 2014.FinleyAO, DattaA, CookBC, MortonDC, AndersenHE, and BanerjeeS. Efficient algorithms for Bayesian Nearest Neighbor Gaussian Processes. Journal of Computational and Graphical Statistics, 28(2):401–414, 2019.31543693 Frühwirth-SchnatterS. Finite Mixture and Markov Switching Models. Springer New York, 2006.GamermanD and LopesHF. Markov chain Monte Carlo. Chapman & Hall/CRC Texts in Statistical Science. Chapman & Hall/CRC, Philadelphia, PA, 2 edition, May 2006.GamermanD, dos SantosTR, and FrancoGC. A non-Gaussian family of state-space models with exact marginal likelihood. Journal of Time Series Analysis, 34(6):625–645, 2013.GelY, RafteryAE, and GneitingT. Calibrated probabilistic mesoscale weather field forecasting. Journal of the American Statistical Association, 99(467):575–583, 2004. doi: 10.1198/016214504000000872. URL https://doi.org/10.1198/016214504000000872.GelfandAE and GhoshSK. Model Choice: A Minimum Posterior Predictive Loss Approach. Biometrika, 85(1):1–11, June 1998.GramacyRB and ApleyDW. Local Gaussian process approximation for large computer experiments. Journal of Computational and Graphical Statistics, 24(2):561–578, 2015. doi: 10.1080/10618600.2014.914442. URL http://dx.doi.org/10.1080/10618600.2014.914442.GramacyRB and LeeHKH. Bayesian treed Gaussian process models with an application to computer modeling. Journal of the American Statistical Association, 103 (483):1119–1130, 2008.GuM. Robustcalibration: Robust calibration of computer models in r. arXiv preprint arXiv:2201.01476, 2022.GuM and BergerJO. Parallel partial Gaussian process emulation for computer models with massive output. The Annals of Applied Statistics, pages 1317–1347, 2016.GuM and LiH. Gaussian orthogonal latent factor processes for large incomplete matrices of correlated data. Bayesian Analysis, 17(4):1219–1244, 2022.GuM and ShenW. Generalized probabilistic principal component analysis of correlated data. Journal of Machine Learning Research, 21(13):1–41, 2020.34305477 GuM, PalomoJ, and BergerJO. Robustgasp: Robust Gaussian stochastic process emulation in r. R Journal, 11(1), 2019.GuhaniyogiR and BanerjeeS. Meta-kriging: Scalable Bayesian modeling and inference for massive spatial datasets. Technometrics, 60(4):430–444, 2018.31007296 GuhaniyogiR, LiC, SavitskyTD, and SrivastavaS. A divide-and-conquer Bayesian approach to large-scale kriging. arXiv preprint arXiv:1712.09767, 2017.GuptaAK and NagarDK. Matrix variate distributions. CRC Press, 1999.HaylockR and O’HaganA. Bayesian Statistics, volume 5. Oxford University Press, New York, 1996.HeatonMJ, DattaA, FinleyAO, FurrerR, GuinnessJ, GuhaniyogiR, GerberF, GramacyRB, HammerlingDM, KatzfussM, LindgrenF, NychkaDW, SunF, and Zammit-MangionA. A case study competition among methods for analyzing large spatial data. Journal of Agricultural, Biological, and Environmental Statistics, 24(3):398–425, 2019.31496633 HewittCG. Conservation of wild life in canada. Nature, 99:246–247, 1917.HigdonD, KennedyM, CavendishJC, CafeoJA, and RyneRD. Combining fielddata and computer simulations for calibration and prediction. SIAM Journal on Scientific Computing, 26(2):448–466, 2004.HigdonD, GattikerJ, WilliamsB, and RightleyM. Computer model calibration using high-dimensional output. Journal of the American Statistical Association, 103(482): 570–583, 2008.HirtM and DellaportasP. Scalable Bayesian learning for state space models using variational inference with smc samplers. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 76–86. PMLR, 2019.KangX and DengX. Design and analysis of computer experiments with quantitative and qualitative inputs: A selective review. WIREs Data Mining and Knowledge Discovery, 10(3):e1358, 2020. doi: 10.1002/widm.1358. URL https://wires.onlinelibrary.wiley.com/doi/abs/10.1002/widm.1358.KatzfussM. A multi-resolution approximation for massive spatial datasets. Journal of the American Statistical Association, 112:201–214, 2017. doi: 10.1080/01621459.2015.1123632. URL http://dx.doi.org/10.1080/01621459.2015.1123632.KatzfussM and GuinnessJ. A general framework for Vecchia approximations of Gaussian processes. Statistical Science, 36(1):124–141, 2021. doi: 10.1214/19-STS755. URL https://doi.org/10.1214/19-STS755.KeelingMJ and RohaniP. Modeling Infectious Diseases in Humans and Animals. Princeton University Press, 2008.KennedyMC and O’HaganA. Bayesian calibration of computer models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(3):425–464, 2001.KrockML, KleiberW, HammerlingD, and BeckerS. Modeling massive highly multivariate nonstationary spatial data with the basis graphical lasso. Journal of Computational and Graphical Statistics, 0(0):1–16, 2023. doi: 10.1080/10618600.2023.2174126. URL https://doi.org/10.1080/10618600.2023.2174126.LiuF and WestM. A Dynamic Modelling Strategy for Bayesian Computer Model Emulation. Bayesian Analysis, 4(2):393–412, 2009.MerrillE, FernA, FernX, and DolatniaN. An empirical study of Bayesian optimization: acquisition versus partition. The Journal of Machine Learning Research, 22(1):200–224, 2021.NobleJV. Geographic and temporal development of plagues. Nature, 250(5469):726–729, 1974.4606583 NychkaD, BandyopadhyayS, HammerlingD, LindgrenF, and SainS. A multiresolution Gaussian process model for the analysis of large spatial datasets. Journal of Computational and Graphical Statistics, 24(2):579–599, 2015.OakleyJE and O’HaganA. Probabilistic sensitivity analysis of complex models: a Bayesian approach. Journal of the Royal Statistical Society Series B: Statistical Methodology, 66(3): 751–769, 2004.O’HaganA. Bayesian Statistics, volume 4. Oxford University Press, New York, 1992.PanS, ZhangL, BradleyJR, and BanerjeeS. Bayesian inference for spatial-temporal non-Gaussian data using predictive stacking, 2024. URL https://arxiv.org/abs/2406.04655.PeruzziM, BanerjeeS, and FinleyA. Highly Scalable Bayesian Geostatistical Modeling via Meshed Gaussian Processes on Partitioned Domains. Journal of the American Statistical Association, 117(538):969–982, 2022.35935897 PetrisG, PetroneS, and CampagnoliP. Dynamic Linear Models with R. Springer New York, 2009.PooleD and RafteryAE. Inference for deterministic simulation models: the Bayesian melding approach. Journal of the American Statistical Association, 95(452):1244–1255, 2000.PresicceL and BanerjeeS. Bayesian transfer learning for artificially intelligent geospatial systems: A predictive stacking approach. arXiv preprint arXiv:2410.09504, 2024.Quinonero-CandelaJ and RasmussenCE. A unifying view of sparse approximate Gaussian process regression. The Journal of Machine Learning Research, 6:1939–1959, 2005.RadevST, MertensUK, VossA, ArdizzoneL, and KötheU. Bayesflow: Learning complex stochastic models with invertible neural networks. IEEE Tranacttions on Neural Networks ad Learning Systems, 33(4):1452–1466, 2022.RasmussenCE and WilliamsCKI. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.RauchHE, TungF, and StriebelCT. Maximum likelihood estimates of linear dynamic systems. AIAA journal, 3(8):1445–1450, 1965.RenQ and BanerjeeS. Hierarchical factor models for large spatially misaligned data: A low-rank predictive process approach. Biometrics, 69(1):19–30, 2013. doi: 10.1111/j.1541-0420.2012.01832.x. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1541-0420.2012.01832.x.23379832 RenQ, BanerjeeS, FinleyAO, and HodgesJS. Variational Bayesian methods for spatial data analysis. Computational Statistics & Data Analysis, 55(12):3197–3217, 2011. ISSN 0167–9473. doi: 10.1016/j.csda.2011.05.021. URL https://www.sciencedirect.com/science/article/pii/S0167947311002003.RobertC and CasellaG. Monte Carlo statistical methods. Springer Texts in Statistics. Springer, New York, NY, 2 edition, Jul 2005.RueH, MartinoS, and ChopinN. Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 71(2):319–392, 2009.SacksJ, WelchWJ, MitchellTJ, and WynnHP. Design and Analysis of Computer Experiments. Statistical Science, 4(4):409 – 423, 1989. doi: 10.1214/ss/1177012413. URL https://doi.org/10.1214/ss/1177012413.Sainsbury-DaleM, Zammit-MangionA, and HuserR. Likelihood-free parameter estimation with neural bayes estimators. The American Statistician, 78(1):1–14, 2024.SantnerTJ, WilliamsBJ, and NotzWI. The design and analysis of computer experiments the design and analysis of computer experiments. Springer series in statistics. Springer, New York, NY, 2 edition, Jan. 2019.SärkkäS and SvenssonL. Bayesian filtering and smoothing, volume 17. Cambridge university press, 2013.SauerA, CooperA, and GramacyRB. Vecchia-approximated deep Gaussian processes for computer experiments. Journal of Computational and Graphical Statistics, 32(3):824–837, 2023a. doi: 10.1080/10618600.2022.2129662. URL https://doi.org/10.1080/10618600.2022.2129662.SauerA, GramacyRB, and HigdonD. Active learning for deep Gaussian process surrogates. Technometrics, 65(1):4–18, 2023b. doi: 10.1080/00401706.2021.2008505. URL https://doi.org/10.1080/00401706.2021.2008505.SiewC. spreadr: An R package to simulate spreading activation in a network. Behavior Research Methods, 2019.SnelsonE and GhahramaniZ. Sparse Gaussian processes using pseudo-inputs. Advances in neural information processing systems, 18, 2005.SoetaertK, PetzoldtT, and SetzerRW. Solving di erential equations in r: Package desolve. Journal of Statistical Software, 33(9), 2010.StroudJR, SteinML, BarryDJS, LeshtM, and BeletskyD. An ensemble kalman filter and smoother for satellite data assimilation. Journal of the American Statistical Association, 105(491):978–990, 2010. doi: 10.1198/jasa.2010.ap07636. URL https://doi.org/10.1198/jasa.2010.ap07636.TurnerR, DeisenrothM, and RasmussenC. State-Space Inference and Learning with Gaussian Processes. In TehYW and TitteringtonM, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, 2010.Vega YonG and ValenteT. netdiffuser: Analysis of diffusion and contagion processes on networks. R package version, 1(3), 2021.VehtariA, GelmanA, and GabryJ. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistical Computing, 27:1413–1432, 2017.VenkitaramanA, ChatterjeeS, and HandelP. Gaussian processes over graphs. In ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5640–5644. IEEE, 2020.VitevitchM, ErcalG, and AdagarlaB. Simulating Retrieval from a Highly Clustered Network: Implications for Spoken Word Recognition. Frontiers in psychology, 2011.WangJ, CockayneJ, ChkrebtiiO, SullivanT, and OatesC. Bayesian numerical methods for nonlinear partial differential equations. Statistics and Computing, 2021.WattsDJ and StrogatzSH. Collective dynamics of ‘small-world’networks. Nature, 393 (6684):440–442, 1998.9623998 WelchWJ, BuckRJ, SacksJ, WynnHP, MitchellTJ, and MorrisMD. Screening, predicting, and computer experiments. Technometrics, 34(1):15–25, 1992.WestM and HarrisonJ. Bayesian Forecasting and Dynamic Models (2nd Ed.). Springer-Verlag, 1997.WestM, HarrisonPJ, and MigonHS. Dynamic generalized linear models and Bayesian forecasting. Journal of the American Statistical Association, 80(389):73–83, 1985.WikleCK. Hierarchical Bayesian models for predicting the spread of ecological processes. Ecology, 84(6):1382–1394, 2003.WikleCK. Low-rank representations for spatial processes. Handbook of Spatial Statistics, pages 107–118, 2010. GelfandAE, DiggleP, FuentesM. and GuttorpP, editors, Chapman and Hall/CRC, pp. 107–118.WikleCK and BerlinerLM. A Bayesian tutorial for data assimilation. Physica D: Nonlinear Phenomena, 230(1):1–16, 2007. ISSN 0167–2789. doi: 10.1016/j.physd.2006.09.017. URL https://www.sciencedirect.com/science/article/pii/S016727890600354X. Data Assimilation.WikleCK and HootenMB. A general science-based framework for dynamical spatiotemporal models. TEST, 19(3):417–451, Nov 2010. ISSN 1863–8260. doi: 10.1007/s11749-010-0209-z. URL https://doi.org/10.1007/s11749-010-0209-z.WilsonA and NickischH. Kernel interpolation for scalable structured Gaussian processes (kiss-gp). In International conference on machine learning, pages 1775–1784. PMLR, 2015.Zammit-MangionA, Sainsbury-DaleM, and HuserR. Neural methods for amortized inference. Annual Review of Statistics and Its Application, 12, 2024.ZhangL and BanerjeeS. Spatial factor modeling: A Bayesian matrix-normal approach for misaligned data. Biometrics, 78(2):560–573, June 2022.33704776 ZhangL, DattaA, and BanerjeeS. Practical Bayesian modeling and inference for massive spatial data sets on modest computing environments. Statistical Analysis and Data Mining: The ASA Data Science Journal, 12(3):197–209, 2019.ZhangL, TangW, and BanerjeeS. Bayesian geostatistics using predictive stacking. arXiv:2304.12414v2, 2024. URL https://arxiv.org/abs/2304.12414.ZhangT and WangW. Existence of traveling wave solutions for influenza model with treatment. Journal of Mathematical Analysis and Applications, 419(1):469–495, 2014.ZhuS, YuK, and GongY. Predictive matrix-variate t models. In PlattJ, KollerD, SingerY, and RoweisS, editors, Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007. URL https://proceedings.neurips.cc/paper_files/paper/2007/file/061412e4a03c02f9902576ec55ebbe77-Paper.pdf.

A graphical illustration of the emulation/calibration paradigm. A statistical model emulates the mechanistic system from experimental runs to enable efficient calibration based on observed data. We build the emulator and calibrator modules through a combination of Gaussian and non-Gaussian state-space methods and Gaussian process regression.

The transfer learning scenario. At each time point, a χ×S matrix is recorded. Within each Yt, information propagates from subdomains within the grid 𝒟1t∪⋯∪𝒟Kt, before propagating into subdomains for the next matrix Yt+1.

Posterior distributions of the variances of the prey (left), the predator (right), and the correlation between the two populations (center). The 95% credible interval of the correlation is (−0.179, −0.056), capturing the negative relation between the (log-transformed) predator and prey populations.

Prey and predator log-populations are plotted against their predicted values with 95% credible intervals for data corresponding to inputs held out of the training phase. The top three plots contain the point estimates with their credible intervals (turquoise) with the ground truth curves in red. The middle three plots correspond to the log-prey populations and the bottom three correspond to the log-predator populations. The black diagonal line denotes the region where the predicted and true values are equal. Each column corresponds to a distinct value of η.

WAIC over episode/time for FFBS models with three different variance structures: inverse-Wishart, inverse-Gamma with spatial correlation matrix R, and I.

Spatiotemporal dynamics of coupled nonlinear partial differential equations. Darker colors indicate larger function values. Wave-like dynamics radiating outwards from the initial seeding locations are apparent.

Heatmap comparison of one example of spatiotemporal fields between PDE solutions and FFBS emulations. The parameters are η1=3.549,η2=0.268,α1=0.010,α2=0.143, and α3=0.170 at times t=18,31,43. The first row presents the space-time fields generated by the PDE, while the second row estimates the field for some unobserved inputs held out of the training data.

Scatter plot featuring a 95% credible interval that contrasts PDE solutions and FFBS emulation results corresponding to some unobserved inputs held out of training data. The red line denotes the region where emulation is perfect.

The simulated trajectories for 50 different parameter values of the Lotka-Volterra equations (colored splines) along with the ground-truth log-populations of hares (left) and lynxes (right). The populations of the hares and lynxes are expressed in the points in each plot.

20,000 samples of η drawn with the calibrator on real Canadian lynx and snowshoe hare population data. The black vertical line denotes the median of the samples, while the dotted lines denote the 2.5th and 97.5th percentiles.

Histograms of posterior densities for elements of parameters η. The red solid line represents the true value, the black solid line marks the median of the MCMC samples, and the black dotted lines indicate 95% credible intervals.

Heat maps of the spatial field generated by the PDE at t = 3, 12, 21. The first row is generated from the true η, while the second and third rows are generated from the 2.5% and 97.5% quantiles of posterior samples of η.

Joint visualization of posterior variability, spatial bias, and field predictions in calibration models, showing the median (blue dotted line and scatter points) with 95% credible intervals (light blue bands). Red lines indicate a perfect match.

Individual node dynamics across the network. Each line corresponds to an activation level for a specific node over time.

Network diffusion calibration posteriors. The lower uncertainty in the hierarchical model indicates pooling of information across node locations, lost in the heterogeneous model. Posterior mode point estimates are roughly similar.

The WAIC and its component statistics computed from the PDE emulation data with different variance structure Σ including inverse-Wishart, σ2R, and I.

Σ = inverse-WishartΣ = σ2RΣ = I (reference)
lppd77.8 × 10655.7 × 106−17.1 × 106
lppd (analytic computation)79.3 × 10657.1 × 106−16.2 × 106
p WAIC 5.2 × 1064.7 × 1063.2 × 106
WAIC = −2(lppd − pWAIC)−145.4 × 106101.9 × 10640.6 × 106

The model comparison with independent replicates. Mirroring the results from Table 1, the inverse-Wishart Σ has the best fit, followed by Σ = σ2R, and then by Σ = I.

Σ = inverse-WishartΣ = σ2RΣ = I (reference)
G 3.0 × 1043.0 × 1043.0 × 104
P 1.9 × 1073.0 × 107935.0 × 107
D = G + P1.9 × 1073.0 × 107935.0 × 107