I am looking to apply the concept of type M and S errors to a network meta-analysis I’m working on, hoping to use the following function developed by Andrew

retrodesign <- function(A, s, alpha=.05, df=Inf, n.sims=10000) { 
  z <- qt(1-alpha/2, df)
  p.hi <- 1 - pt(z-A/s, df)
  p.lo <- pt(-z-A/s, df)  
  power <- p.hi + p.lo
  typeS <- p.lo/power
  estimate <- A + s*rt(n.sims,df)
  significant <- abs(estimate) > s*z
  exaggeration <- mean(abs(estimate)[significant])/A
              typeS=typeS, exaggeration=exaggeration))

Where A is your hypothesized true effect and s is the standard error of your estimate. The code for NMA calculates basic parameters which in my case is the mean difference between each active treatment and placebo. These are then used to calculate all the remaining pairwise comparisons. My question is an embarassingly simple one, and is really just meant to make sure I am not making a huge error:

Am I right to calculate the standard error for a comparison based on the SD of the mcmc results for that parameter (e.g. sd/sqrt(n))? Or do I just use the SD of the estimate in lieu of the standard error?


The standard error is just the standard deviation of the parameter samples, but you may want to make sure that's what you want to use. Since you're conducting Bayesian analysis you shouldn't do classical hypothesis testing with your results (since those tests are meaningless in this context). Instead, use the sample of parameters to perform Bayesian hypothesis testing, which is actually much easier and more intuitive.

As an example, if you're wanting to find the probability a parameter $\theta$ is positive (i.e., $\Pr(\theta>0|D)$), simply calculate $$\frac1N \sum_{i=1}^N I(\theta^{[i]} > 0),$$ where $N$ is the number of sampling iterations, $\theta^{[i]}$ is the $i$th sample of $\theta$, and $I$ is an indicator function. In R, this is equivalent to sum(theta > 0) / N.

This is one of the main benefits of Bayesian analysis (which comes at the "cost" of needing to specify priors over variables). Rather than relying on asymptotic distributions of parameters to perform hypothesis tests that may not be a good approximation, you estimate the actual distributions of parameters (which depend on prior assumptions and the data) and use those to conduct inference directly.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.