Can prior distributions incorporate uncertainty AND variability in parameters?

Question

I am trying to understand how to interpret Bayesian prior and posterior distributions in situations where there is believed to be variability in model parameters (due to variation in the population under study) as well as uncertainty. My understanding is that, typically, the prior distribution for a model parameter represents uncertainty in our knowledge of that parameter. My question is, would it be statistically valid to proceed with the computation of the posterior using priors that incorporate both uncertainty and variability? And would the interpretation of the posterior need to change if this was done?

Thoughts:

1. I am aware that, if a parameter is believed to vary, then it would be possible to specify a hierarchical model, with variability in the parameter quantified by a hyperparameter (and uncertainty in this hyperparameter represented by an associated hyperprior). So I guess this is one approach.
2. In the context of quantitative risk analysis (QRA e.g. in books by Vose) parameter distributions can be chosen to represent uncertainty and variability. These distributions seem very similar to prior distributions. The difference being that, usually, no posterior distribution is computed - instead, something akin to a prior predictive distribution would typically be computed.

Answer 1

Consider that you measure $x_{i,j}$ in site $i$ according to ($\sigma_i$ being considered here as known and $\mu_i$ unknown being your parameter value for site $i$): $$ p(x_{i,j}|\mu_i,\sigma_i) $$ You know that the parameter $\mu_i$ vary from one site $i$ to the other but consider as reasonable to represent this variation by a density $$ p(\mu_i|\sigma,\mu) $$ with scale and location parameters $\mu$ and $\sigma$ unknown. Then you can perfectly estimate $$ p(\mu,\sigma|(x_{i,j})_{i,j}) $$ or $$ p((\mu_i)|(x_{i,j})_{i,j}) $$ that both account for variation around $\mu$ and measurement error around $\mu_i$. Notice that if you assume a normal distribution for the two above-mentioned densities you have: $$ X_{i,j}|\mu,\sigma,\sigma_i \sim N(\mu,\sigma^2+\sigma^2_i) $$ where you can consider that $\sigma$ represents your inter-site variability and $\sigma_i$, your uncertainty related to the measurement process. Moreover even with a limited observation number $J$ for each site ($j$ in $1:J$), the accuracy of $\mu$ and $\sigma$ estimates will increase when the sample number $I$ increases.

Answer 2

There are priors that encode variability. However, these can't be priors over the parameters of a distribution of an individual example. These priors have to involve multiple examples. For one example of encouraging variability in latent features, see Wang, Ye, and David B. Dunson. "Probabilistic Curve Learning: Coulomb Repulsion and the Electrostatic Gaussian Process." Advances in Neural Information Processing Systems. 2015.

Here's a simple exampe: If the $x_i$ are your data points, then a likelihood term $$\sigma\left(\frac{(x_i-x_j)^2}{\theta}\right) \qquad \forall i\ne j$$ where $\sigma$ is the logistic sigmoid, encourages variability. A large $\theta$ pushes the $x_i$s apart. You can still encode your uncertainty by having wide priors on the $x_i$. Note how this constraint is over pairs of examples — a constraint over an individual example cannot enforce variability.