Consider spatially distributed data $\boldsymbol x = \{x(t_1),x(t_2),\dots,x(t_n)\}$ which is described by a multivariate Gaussian distribution with mean $\mu$, standard deviation $\sigma$ and stationary correlation function which depends on the absolute distance $\tau = |t_1-t_2|$ between points:

$$\rho(\tau) = \exp\left(-\frac{2\tau}{\theta}\right)$$

Which produces a correlation matrix $\boldsymbol \rho(\theta)$.

$$\boldsymbol x \sim \mathcal{N}(\mu,\sigma^2 \boldsymbol \rho)$$

Given samples of $\boldsymbol x$ the aim is to predict $\mu$, $\sigma$ and $\theta$. Under the Bayesian approach I believe that a joint prior distribution must be specified. Since I want to use Markov Chain Monte Carlo to sample from the posterior I believe the prior should be proper. My approach is to specify marginal distributions $P(\mu)$, $P(\sigma)$, $P(\theta)$ and take the joint prior to be given by:

$$P(\mu,\sigma,\theta) = P(\mu)P(\sigma)P(\theta)$$

Is it reasonable to take the priors to be independent?

Can I take the priors to be uniform if am confident the parameter values fall in certain intervals i.e.

$$\mu \sim U[-100,100]$$ $$\sigma \sim U[0,1000]$$ $$\theta \sim U[0,1000]$$

Or should I use a Cauchy for $\mu$ and the folded-Cauchy for $\sigma$, $\theta$?

  • $\begingroup$ What's this for, kriging? $\endgroup$ – Jon Feb 2 '17 at 18:46
  • $\begingroup$ Often a inverse Gamma prior is assumed for $\sigma$. Do you have any specific reasons (read: prior knowledge/belief) to prefer a uniform prior? $\endgroup$ – DeltaIV Feb 2 '17 at 20:41
  • $\begingroup$ @DeltaIV Jarad's comment has made me realise that a uniform prior is not non-informative. Do I believe that $Pr(0 \leq \sigma \leq 100) = Pr(900 \leq \sigma \leq 1000)?$. No. However I don't like inverse Gamma due to its behaviour around zero (see here). I think I will go with the folded Cauchy. $\endgroup$ – Fundamental Engineer Feb 2 '17 at 20:45
  • $\begingroup$ Any prior can be used, there is no way one can recommend one prior as been better than another on a general basis. $\endgroup$ – Xi'an Feb 2 '17 at 21:39

I believe you are using uniform priors in an attempt to be non-informative, but I believe your priors are more informative than you would like them to be. Do you really believe $\sigma$ or ($\theta$) is equally likely to be between (0,1) as (999,1000)? And that there is absolutely no probability whatsoever that these parameters are above 1000?

Also, there is no requirement to have proper priors for MCMC.

My suggestion is $\sigma \sim Ca^+(0,c_\sigma)$ (i.e. a folded-Cauchy or half-Cauchy), $\theta \sim Ca^+(0,c_\theta)$, an improper uniform for $\mu$ and all independent. You will need to choose $c_\sigma$ and $c_\theta$ for your problem, but since you are already "confident" the parameter values fall into the regions given, you can just choose these values so the probability outside those regions is small. I prefer heavy-tailed distributions like the half-Cauchy because the data can more easily overwhelm the prior in case you were wrong in your prior assumptions. If you really want the prior to be proper, you could use a Cauchy for $\mu$ and specify a location and scale parameter.

  • $\begingroup$ Thanks Jarad. I know that proper posteriors can come from proper priors thanks to your youtube video. I am not against improper priors per se however I want to ensure my posterior is proper. How do we know that the improper uniform prior will yield a proper posterior? $\endgroup$ – Fundamental Engineer Feb 2 '17 at 20:41
  • $\begingroup$ This is a question that need be solved on a case-by-case basis since you have to integrate out the likelihood against the prior to demonstrate properness. A computational approximate resolution is to check for stability of the posterior as a sequence of proper priors converging to the improper prior grows. $\endgroup$ – Xi'an Feb 2 '17 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.