Why doesn't it seem to be standard to multiply prior scale by fraction of non-zero predictor observations?

Question

When doing multivariate regression, it's often the case that some predictors often have many zero values - dichotomous inputs, dummy coding of polychotomous inputs, interval coding, etc. The fraction of nonzero observations for a predictor is especially decreased when interacting these variables.

It seems obvious that only the observations of a predictor for which it is nonzero (after any transformations) will affect the estimation of its coefficient.

When doing multivariate regression I like to use weakly informative priors (like Gaussian or Cauchy) for regularization. I've noticed that sometimes, predictors which are usually zero still take on implausible regression coefficients, like ~6 for a logistic regression model. This seems to be because a variable that is nonzero for only a handful of observations is still overcoming the prior because it's getting full credit for all of its zero observations, and thus getting a huge coefficient, which is in many cases a prior obviously overfitting, and hurts generalization performance for prediction.

Multiplying the prior's scale by the fraction of observations of a predictor that are nonzero fixes this and greatly improves generalization performance, makes coefficients plausible, and seems to be common sense.

So - why don't more people do this? I've tried a cursory search in common textbooks (various machine learning books, applied and theoretical Bayesian statistics), R packages on CRAN, some machine learning packages, and I don't see evidence of anyone else doing this... isn't this just common sense? Or is there something wrong with this approach that I'm missing?

Answer 1

It is difficult to understand quite what you are saying here and it would be helpful to have an illustration. Take for example the values $x_1$, $x_2$ and $y$ here

x_1  x_2   y
---  ---  ---
 1  1     8
-1  1    -1
-3  0   -15
 2  0     9
 0  0    -1
 2  0    10
-1  0    -6

OLS regression would give $\hat{y} = 4.9 x_1 + 4.1 x_2 - 0.6$. I don't see how the $+4.1$ can be said to be implausible. What is true is that it is a more uncertain estimate than the estimate of $+4.9$ since there is less information about the impact of variation of $x_2$ than about the impact of variation of $x_1$. [When I generated these numbers, I was aiming for $5 x_1 + 3 x_2$.]

This is not the same as saying $x_2$ has too many zeros. Let $x_3 = 1 - x_2$, so $x_3$ is zero only twice, rather than five times. Then OLS regression of $y$ on $x_1$ and $x_3$ would give $\hat{y} = 4.9 x_1 - 4.1 x_3 +3.5$, much as you might expect. But the new coefficient of $-4.1$ is as good (in any sense) as the earlier $+4.1$ even though there are fewer zeros.

On a different issue, if you are using a prior distribution which is not scale-free and you have other prior information that suggests to you that the dispersion of plausible values is going to be narrower or wider than suggested by your distribution then you can (indeed you should) adjust your prior distribution accordingly.