When we have sparse data and need to regularize the posterior, we could use informative priors instead of vague ones.

However, the information that we choose to construct those informative priors is sometimes not directly related (i.e. it might pertain to a parameter that we know is only similar but not the same with model parameter).

My question is "what is exactly the assumption that we are making when we use some information in order to construct an informative prior that will regularize the estimation of our sparse data?"

(Unfortunately i cannot provide a particular example since I am only theoretically trying to understand the implications)

  • $\begingroup$ Since every prior is informative, what exactly is your question? The prior itself is the assumption about the parameter. $\endgroup$
    – Tim
    Feb 13, 2018 at 16:30
  • $\begingroup$ Do you really mean assumption? Is "interpretation" a better term? It is a relative concept: priors summarize belief. $\endgroup$
    – AdamO
    Feb 13, 2018 at 18:27

3 Answers 3


When you are using a vague prior, you are assuming that a priori, your parameter is equally likely to take most admissible values. When you are using an informative prior, you are assuming that a priori, your parameter is significantly more likely to take particular values than others.

The simple answer is: The assumptions are what the prior encodes.

It is difficult to explain this more clearly without circular logic unless there's an example, so since you don't provide one, I'll make one up myself. Let's say that you are doing a regression, and you have regressors $x_1$ and $x_2$ and target $y$. The model is a regular old Gaussian model, $$ p(y|a_0,a_1, a_2) = \mathcal{N}(y|a_0 + a_1 x_1 + a_2 x_2, 1). $$ Furthermore let's say that $x_1$, $x_2$ are binary, and $|y|<10$ usually (in order to get a reasonable scaling). What kind of prior for $a_0, a_1, a_2$ is reasonable to use?

A vague prior would set $p(a_i) = \mathcal{N}(a_i|0, 100^2)$, for instance, because this prior is essentially flat for reasonable values of the coefficients, in light of the scaling of the data; it will probably not affect the posterior inference much if we set the prior variance to $1000^2$ either.

However, let's say that we don't think that all $a_i$ are equally likely to be $0$ or $5$ or $10$.

  • There might be a priori reasons to suspect that the regressor $x_2$ doesn't have much effect on $y$. If we wanted to implement this prior knowledge into the prior distribution, we might pick $p(a_2) = \mathcal{N}(a_2|0, 1^2)$ instead. This distribution is centered around $0$ and has low variance, so it should constrain $a_2$ to be close to $0$ unless the data "overrules" it strongly.

  • Our prior suspicion for $z_1$ might be that it has a positive effect, but that the magnitude is more uncertain. Thus, we might choose $p(a_1) = \mathcal{N}(a_1|3, 5^2)$. This prior has a majority (70%) of its probability mass at $a_1>0$.

  • Finally, maybe we don't know much about the intercept $a_0$; however, because of the scale of the data, it can't be too big. Thus, let's set $p(a_0) = \mathcal{N}(a_0|0, 10^2)$.

To finish up, note that it is important to know that all priors contain some information. The question is how much information they contain. Even the vague priors above contained some information in order to ensure that $a_i$ could not, for example, have order of magnitude 10^10. However, with the informative prior designed above, we added our prior knowledge into the statistical model.


I interpret the question to be about learning across "units." Here is a simple framework that can be used to illustrate the ideas involved.

Consider two units, labeled 1 and 2. Let the data applicable to unit $i$ be denoted $y_i$ (where $y_i$ may be composed of a number of observations.) Assume the sampling distributions are conditionally independent: \begin{equation} p(y_1|\theta_1)\,p(y_2|\theta_2) . \end{equation} Suppose we have the posterior distribution for $\theta_1$ based on $y_1$: \begin{equation} p(\theta_1|y_1) = \frac{p(y_1|\theta_1)\,p(\theta_1)}{p(y_1)} . \end{equation} The question is this: How can we use the information about $\theta_1$ to provide a prior for $\theta_2$?

We can do this through dependence in the joint prior for $\theta_1$ and $\theta_2$: \begin{equation} p(\theta_1,\theta_2) = p(\theta_2|\theta_1)\,p(\theta_1) . \end{equation} In particular, \begin{equation} p(\theta_2|y_1) = \int p(\theta_2|\theta_1)\,p(\theta_1|y_1)\,d\theta_1 . \end{equation} We now have a prior for $\theta_2$ that is informed by non-sample information (i.e., not $y_2$). With this prior can can compute \begin{equation} p(\theta_2|y_1,y_2) = \frac{p(y_2|\theta_2)\,p(\theta_2|y_1)}{p(y_2|y_1)} . \end{equation} If more than one other unit were available, then an even more informed prior for $\theta_2$ could be computed.

Of course there remains the question about how to structure the prior dependence between $\theta_1$ and $\theta_2$. Hierarchical models provide a convenient way to do this. Suppose $\theta_1$ and $\theta_2$ are conditionally independent given the hyperparameter $\phi$: \begin{equation} p(\theta_1,\theta_2|\phi) = p(\theta_1|\phi)\,p(\theta_2|\phi) . \end{equation} The hyperparameter is unknown and its prior is $p(\phi)$. With this structure we have \begin{equation} p(\theta_1,\phi|y_1) = \frac{p(y_1|\theta_1)\,p(\theta_1|\phi)\,p(\phi)}{p(y_1)} \end{equation} and \begin{equation} p(\theta_2|y_1) = \int p(\theta_2|\phi)\, p(\phi|y_1)\,d\phi . \end{equation} In this formulation the hyperparameter can be interpreted as the carrier of the information.

This is just a very brief sketch. More can be said at every stage.


My answer is similar to Martin's. Let's look at an example: Ridge (linear) regression :say for $y=\beta X+\epsilon$. The motivation is to reduce the variance: avoid over-fitting. Imagine input and output are normalized so that what I'm going to say is not sensitive to the scale.

The prior on $\beta$ is $N(0,b^2I)$? The assumption is "$\beta$ can't have a big norm". How much is "big"? By definition, that's the regularization parameter $b$. Since the data is normalized "big norm" means both a lot of big positives and big negative coefficients.

The choice of $b$ does not matter much. The method is not much sensitive to the precise value of $b$ but just needs to be in "right range". The curve $b\rightarrow error$ have most often a big flat plateau. The range is often determined with a validation set.

The unregularized estimate is over-fitted and produces big (and inaccurate) $\beta$s. Two questions:

  1. why and how much?

I once did a few simulations in willingly extreme scenarios where the regularized version works very well but the norm of the unregularized estimate happens to be thousands or millions of time the true value's with a high probability. The theoretical answer to "how much" is the variance of the raw estimator which can be calculated explicitly as $\sigma^2(X'X)^{-1}$: see wikipedia. Typically near-collinear inputs make this value explode. The more features, the smaller data, the more near-colinearity is likely to happen randomly.

  1. why big $\beta$s could not be the "true" parameter? Can ruling them out make you miss the true value?

Why real life data is often like this?. I don't know. Theoretically everything is possible, but... this is very unlikely to happen. And even if it was possible, you could not approximate the value with your data anyway in a satisfying way. It's a bit like a Pascal's Wager: if $\beta$ has a big norm, you can't find any useful approximation of it, so that you'd better assume it has a small norm, and use a good estimator for this case. So rather than prior knowledge, the prior is more like a bet.

Note: A purely non Bayesian justification of regularization is something I'm still looking for. There are ways to find the best (from some point of view) value of $b$ explicitly: Determination of Tikhonov factor. I'm not sure, but even if not explicitly Bayesian, a Bayesian assumption may be subtly hidden in it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.