Wahba (1983) and Silverman (1985) show that the quadratic penalty term on a smoothing spline is akin to a bayesian prior on the smoothness of the model. Nychka (1988) is another key reference. This is made a little less-arcane by Simon Wood (see section 4.8). Consider the model $$ y = \mathbf{X\beta} + \epsilon $$ where $\mathbf{X} = [x_1, x_2, ..., x_k]$, $x_k = b_k(z)$, $b$ is some basis function like a linear spline, and $k$ can grow arbitrarially large while $z$ is a single variable (i.e.: $N\times 1$).

Because $k$ can be huge, least-squares estimates would badly overfit. The smoothing spline solution is to impose a prior over the size of $\beta$: $$ f_\beta(\beta) \propto e^{-\frac{1}{2} \beta^T D/\tau\beta} $$ here $D$ is the smoother matrix and $\tau$ is some constant. In other words, the model is fit by $$ \hat\beta = (\mathbf{X^TX + D}/\tau)^{-1}\mathbf{X^T}y $$

(This is basically a ridge regression and $\tau$ directly corresponds to $\lambda$!)

It can be shown that this leads to the following posterior distribution for $\beta$: $$ \beta|y \sim \mathcal{N}\left(\hat\beta, (\mathbf{X^TX + D}/\tau \right)^{-1}\sigma^2 $$ Given an estimated penalty parameter chosen by some variant of cross-validation, Wahba's 1983 paper shows that this Bayesian posterior distribution can generate "confidence intervals" with "good frequentist properties" in the sense that if you assume that $y = f(z) + \epsilon$, where $f$ is some fixed function, then intervals generated from that posterior distribution will cover $f$ at the nominal rate. mgcv's confidence bands are built around these ideas.

These ideas were formalized for (generalized) additive models. Do they work for general ridge regression? Specifically, say that I observe some dataset $\mathbf{X}$, and many of its elements are highly redundant. For example, $x_1$ could be the mass of a goat, $x_2$ could be the goat's volume, $x_3$ could be the goat's density, $x_4$ could be the goat's density measured in some different way, $x_5$ could be the mass of the goat's hair, etc. This sort of situation isn't so different from the basis expansion case, except that the statistician doesn't choose the basis expansion -- rather it comes "baked into" the data.

Given this redundancy, a predictive model of goat food consumption as a function of $\mathbf{X}$ will overfit.

It seems natural to impose a penalty therefore on the coefficients, interpret the penalty as a prior on the complexity of the model, and estimate a bayesian covariance matrix as above.

My question:

Will "confidence intervals" (prediction intervals, actually) for ridge regression based on the bayesian covariance matrix have the same frequentist coverage properties as they exhibit in the spline case, as $N\rightarrow$ large???

In other words, is there something specific about the spline case that makes this not work for general ridge regression? Note that I don't care about inference on specific individual parameters, and I'd leave the intercept unpenalized.

The context for the question is that I'm looking to compare a number of forecasting methods, not just in terms of MSE but also in terms of uncertainty representation/interval performance. And I don't have the training to go full-bayesian.

  • $\begingroup$ I just want to mention that the methods are all different. Wahba is referring to what are more commonly called "credible intervals/regions". Silverman is dealing with classical probability density estimation. Wood's text covers additive models (a general form of modeling) which is also a classical and not a Bayesian approach. $\endgroup$ – Michael Chernick Mar 28 '17 at 16:06
  • $\begingroup$ The Silverman paper that I linked is not density estimation, but rather about bivariate nonparametric regression. And the whole basis of the question is this weird property of bayesian analysis to yield something that is useful from a frequentist perspective. @MichaelChernick $\endgroup$ – generic_user Mar 28 '17 at 16:09
  • $\begingroup$ And Wood's section on distributional properites of GAM estimates builds on this logic -- that a bayesian treatment of a basically frequentist procedure yields intervals with nominal coverage. $\endgroup$ – generic_user Mar 28 '17 at 16:14
  • $\begingroup$ I am not sure about Wood's book as I cannot access Section 4.8. It doesn't seem to relate to your italicized question. Also you have made your question very broad by looking for a variety of forecasting methods. It is proper to capitalize Bayesian. $\endgroup$ – Michael Chernick Mar 28 '17 at 16:15
  • $\begingroup$ Potentially-illegal copies of Wood's book are available in pdf by searching for the title. And the context is broad, but the question is fairly specific to ridge regression. I'm basically asking whether there is something special about quadratically-penalized spline models, as distinct from quadratically-penalized models generally. @MichaelChernick $\endgroup$ – generic_user Mar 28 '17 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.