# Framework for simulation study to validate bayesian models

I am looking for a framework that would allow to take JAGS/bugs model and on many sets of simulated data test if there is a bias (or not) in the parameter estimates (the real parameters would be known because the data are simulated). Is there any such framework? I've seen only run.jags.study() in R package runjags, but this is focused on crossvalidation. I need to check the bias in parameter estimates.

Do you know such framework? If not, I am especially interested how to evaluate the bias across the generated data sets. How would you do that?

Thanks in advance!

• Could you please elaborate on what you mean by "framework"? Would it be software, a system of thought, a defined statistical procedure, or something else altogether? Given that JAGS and BUGS explicitly represent probability models, the process of simulating data from them would appear to be without any conceptual difficulties. – whuber Jun 29 '18 at 14:43
• I mean basically anything that would facilitate (not only conceptually, but also practically :)) the whole process of generating and fitting the multiple data sets and, most importantly, evaluating the results (I don't know exactly how to do this - here I am not sure about the concept). – Curious Jun 29 '18 at 14:48

## 1 Answer

Cross-validation is only one potential use of the run.jags.study function - it can also be used for estimating model performance in terms of bias and/or coverage of 95% confidence intervals. For example:

library('runjags')

# Target values:
mu_target <- 0.3
tau_target <- 0.2

N <- 10

# Function to simulate data:
datafun <- function(simulation_number){
y <- rnorm(N, mean=mu_target, sd=1/tau_target^0.5)
return(list(y=y))
}

# Simple model:
model <- '
model{

for(i in 1:N){
y[i] ~ dnorm(mu, tau)
}
#data# N

mu ~ dnorm(0, 10^-6)
tau ~ dgamma(0.01, 0.01)
}'

# Get bias and coverage:
results <- run.jags.study(240, model, datafun, targets=list(mu=mu_target, tau=tau_target), n.cores=6)
results

Average values obtained from a JAGS study with a total of 240 simulations:

Target Av.Median Av.Mean Av.Lower95%CI Av.Upper95%CI Av.Range95%CI Prop.Within95%CI Av.AutoCorr(Lag10) Simulations
mu     0.3   0.30215 0.30247       -1.2433        1.8309        3.0742           0.9375          -0.000946         240
tau    0.2   0.23197 0.25005      0.053889       0.48251       0.42862          0.95833       -0.000085487         240

Average time taken:  0.2 seconds (range: 0.2 seconds - 0.3 seconds)
Average adapt+burnin required:  5000 (range: 5000 - 5000)
Average samples required:  10000 (range: 10000 - 10000)


So there is a small positive bias in mean and a slightly larger positive bias for tau, but both sets of 95% confidence intervals contain the true value approximately 95% of the time.

See also section 3 of the following for a more complete/useful example:

Denwood, M.J. 2016. runjags: An R Package Providing Interface Utilities, Model Templates, Parallel Computing Methods and Additional Distributions for MCMC Models in JAGS. J. Stat. Softw. 71. doi:10.18637/jss.v071.i09.

Matt