What kind of results are there about prior knowledge?

Question

Consider the following two problems:

1. We want to do linear regression, where we know that the true model is actually linear. In the general case, linear regression includes an offset. However, we know, using prior knowledge, that the true model actually has offset 0. So clearly, it is better to fit the model while constraining that offset is 0.

2. We want to estimate a distribution $p(z)$ and find its mode. We estimate $p$ using, let's say, density estimator, to be $\tilde{p}$. Now, again, we know some properties about the mode, such that, for some function $F(z_{mode}) = 0$. Then, (perhaps?) it is better to find the mode of $\tilde{p}(z)$ while constraining $F(z)$ to be $0$, instead of just finding the mode of $\tilde{p}$ without this constraint.

My question is: are there any known results about incorporating this kind of prior knowledge (if it is true)? A result that shows that our solution is better when using the prior knowledge, or so on? I hope it is not too vague. I am interested in reading more about this.

Thanks.

Answer 1

With respect to 1.: as chemometrician, my practical question is: do you really have no offset?

One standard situation that corresponds to your scenario from my field of work is linear (univariate) calibration using the Beer-Lambert-law:

$$ A = \varepsilon l c $$
(here $\varepsilon$ is the absorption coefficient, no error term)

In practice, even if you obey the prerequisites the fit will include an offset (blank value) because reality is not as easy as theory. E.g. because you use a double beam instrument to compensate variability in the source intensity, but then you cannot use the same single cuvette for sample and blank, which can (will) cause an offset.

Of course, you may have noise $\gg$ offset, so you can neglect the offset...

Or you say that the true model does have an offset as physical reasons can be given. But taking into account the influence of the measurement my bet is that it will be pretty hard to find an example system that is really supposed to be linear. (Anyone knows an example?)

With respect to the general question: Depending on the situation, however, a constrained model may be better even if it is only an approximation (keyword: bias variance tradeoff, see e.g. the Elements of statistical learning)

Answer 2

I am surprised nobody said anything like "Form a Bayesian model and put your prior information into a prior distribution" -- that's what I hear pretty much as a knee-jerk reaction a lot of times.

I will leave this up to others, though, and point to other areas of statistics where prior knowledge is explicitly incorporated, although in ways simpler than you have suggested here. In survey statistics, such procedures are known as calibrated estimation (and, in a special case, as generalized regression estimation, GREG). Suppose you know that you have a population in which 30% of people are have blue eyes, 50% of people have brown eyes, and 20% of people have green eyes (I am making this up, of course, but I hope you'd bear with me for this). You took a random sample, and you found that you have 27% blue-eyed, 55% brown-eyed, and what, 18% green eyed. If you are absolutely sure that the deviations from the population figures are solely due to the random nature of the sample (sampling deviations), then you can get better estimates of the characteristics correlated with eye color by reweighting your data: give the blue-eyed guys the sampling weight of 30/27 to make up for a slight deficiency, the brown-eyed, the weight of 50/55, and the green-eyed, 20/18. Such procedure would also call for adjustments of the standard errors, and you will see that they will be lower than the standard $s=\sqrt{\sum_i(x_i-\bar x)^2/(n-1)}$, rightfully so.

As an alternative procedure that produces essentially the same result, you can form the residual differnces 0.27-0.30, 0.55-0.50, 0.18-0.20; form the covariance of your statistic of interest (say average height) with the sample proportions of the eye color (i.e., run a regression of height on eye color); and form the prediction from this regression at the population values of the eye color. It will have (asymptotically) the same properties as the reweighted estimator, and will boast a similar reduction in sampling variance.

An ultimate Bible on this is Sarndal, Swensson and Wretman (1992, 2003).

Answer 3

The general comment is that in the abstract it is usually better to make the model space as small as possible. This is sort of vague and obvious at the same time (we would love the model space to be a singleton set!). "As small as possible" really constitutes a decision problem (basically weighing the losses associated with inconsistent versus inefficient inferences).

From a frequentist perspective you (roughly) enhance the degrees of freedom by estimating fewer parameters, which will give you lower variance estimates, more powerful tests etc. From a Bayesian perspective, if you have $\theta = (\alpha, \beta)$ and are interested in $\alpha$ then you're after the marginal posterior $p(\alpha|Y) = \int p(\alpha, \beta|Y)d\beta$. Integrating over $\beta$ inflates the variance in this distribution, so if you're really confident that $\beta$ takes a fixed value you'll get more efficient inferences by fixing it (assuming you're correct!).

In practice you must also be concerned about parameter constraints complicating inference too, although this is outside the scope of your question...