2
$\begingroup$

I am trying to understand how to interpret Bayesian prior and posterior distributions in situations where there is believed to be variability in model parameters (due to variation in the population under study) as well as uncertainty. My understanding is that, typically, the prior distribution for a model parameter represents uncertainty in our knowledge of that parameter. My question is, would it be statistically valid to proceed with the computation of the posterior using priors that incorporate both uncertainty and variability? And would the interpretation of the posterior need to change if this was done?

Thoughts:

  1. I am aware that, if a parameter is believed to vary, then it would be possible to specify a hierarchical model, with variability in the parameter quantified by a hyperparameter (and uncertainty in this hyperparameter represented by an associated hyperprior). So I guess this is one approach.
  2. In the context of quantitative risk analysis (QRA e.g. in books by Vose) parameter distributions can be chosen to represent uncertainty and variability. These distributions seem very similar to prior distributions. The difference being that, usually, no posterior distribution is computed - instead, something akin to a prior predictive distribution would typically be computed.
$\endgroup$
  • $\begingroup$ What do you mean by variability? Do you mean, changing over time? $\endgroup$ – Matthew Gunn Dec 12 '15 at 19:07
  • $\begingroup$ Not exactly... I was thinking of variability between units in the population under study. A specific example I have been thinking of is in the context of disease transmission models for infectious disease transmission within animals on farms. Parameters controlling disease transmission are likely to be unknown, but are also likely to vary between farms. $\endgroup$ – S. Catterall Reinstate Monica Dec 12 '15 at 19:16
  • $\begingroup$ Sounds like you have an unknown parameter for each farm, and the same prior for each parameter? $\endgroup$ – Matthew Gunn Dec 12 '15 at 19:43
  • $\begingroup$ Currently I just have one unknown parameter which is shared by all farms. $\endgroup$ – S. Catterall Reinstate Monica Dec 12 '15 at 20:16
  • $\begingroup$ I think I understand what you're asking now. So you might something like: (1) Some variable (eg. transmission rate) $b_i$ for $n$ farms (2) $b_i$ are drawn from some distribution $C$ (3) The parameters that describe $C$ are also unknown, but you've got some prior for $C$. $\endgroup$ – Matthew Gunn Dec 12 '15 at 20:31
1
$\begingroup$

Consider that you measure $x_{i,j}$ in site $i$ according to ($\sigma_i$ being considered here as known and $\mu_i$ unknown being your parameter value for site $i$): $$ p(x_{i,j}|\mu_i,\sigma_i) $$ You know that the parameter $\mu_i$ vary from one site $i$ to the other but consider as reasonable to represent this variation by a density $$ p(\mu_i|\sigma,\mu) $$ with scale and location parameters $\mu$ and $\sigma$ unknown. Then you can perfectly estimate $$ p(\mu,\sigma|(x_{i,j})_{i,j}) $$ or $$ p((\mu_i)|(x_{i,j})_{i,j}) $$ that both account for variation around $\mu$ and measurement error around $\mu_i$. Notice that if you assume a normal distribution for the two above-mentioned densities you have: $$ X_{i,j}|\mu,\sigma,\sigma_i \sim N(\mu,\sigma^2+\sigma^2_i) $$ where you can consider that $\sigma$ represents your inter-site variability and $\sigma_i$, your uncertainty related to the measurement process. Moreover even with a limited observation number $J$ for each site ($j$ in $1:J$), the accuracy of $\mu$ and $\sigma$ estimates will increase when the sample number $I$ increases.

$\endgroup$
  • $\begingroup$ In this case, don't $\sigma, \sigma_i$ both contribute to uncertainty? How is the maximum a posteriori solution not simply $x_i=\mu_i$? How does such a model encourage variability? $\endgroup$ – Neil G Dec 14 '15 at 14:19
  • $\begingroup$ @Neil G. IMHO uncertainty and variability is more a lexical problematic than a design one. The MAP is not $m_i=x_i$ because of $p(\mu_i | \sigma, \mu)$ that tends to group more or less the $\mu_i$ (depending on the prior for $\sigma$ and $\mu$)(I think that shrinkage estimator is the keyword to look at). The model encourages variability as much as the data (and the choosen prior) encourages it to do it. $\endgroup$ – peuhp Dec 14 '15 at 14:27
  • $\begingroup$ Your density over the $x_i$ is Gaussian, so how is the MAP solution not simply the mean? The whole point of this question is to encourage variability using a prior — you cannot control the data. I still don't see how your prior encourages variability. $\endgroup$ – Neil G Dec 14 '15 at 14:37
  • $\begingroup$ Sorry I will try to make the post clearer. The point is that I have not defined priors on $\sigma$ and $\mu$ so that it seems to me difficult to discuss the MAP. is it ? Maybe, I miss the question but the model encourages variability by allowing it then the data speaks... Thanks for helping me improving the answers :) $\endgroup$ – peuhp Dec 14 '15 at 14:43
  • $\begingroup$ "the model encourages variability by allowing it then the data speaks..." — that is exactly "uncertainty". That is not encouraging variability. By your logic both $\sigma$ and $\sigma_i$ do the same thing: they both encode uncertainty. $\endgroup$ – Neil G Dec 14 '15 at 14:51
-1
$\begingroup$

There are priors that encode variability. However, these can't be priors over the parameters of a distribution of an individual example. These priors have to involve multiple examples. For one example of encouraging variability in latent features, see Wang, Ye, and David B. Dunson. "Probabilistic Curve Learning: Coulomb Repulsion and the Electrostatic Gaussian Process." Advances in Neural Information Processing Systems. 2015.

Here's a simple exampe: If the $x_i$ are your data points, then a likelihood term $$\sigma\left(\frac{(x_i-x_j)^2}{\theta}\right) \qquad \forall i\ne j$$ where $\sigma$ is the logistic sigmoid, encourages variability. A large $\theta$ pushes the $x_i$s apart. You can still encode your uncertainty by having wide priors on the $x_i$. Note how this constraint is over pairs of examples — a constraint over an individual example cannot enforce variability.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.