I have a ridge regression model

$ y = \beta_1 x_1 + \beta_2 x_2 + ...$

The $x$s are highly collinear but are all physically relevant, hence use of ridge regression.

And am considering replacing it with Bayesian MCMC estimation to allow use of a more complex model

$ y = \delta^{x_0} ( \beta_1 x_1^{\gamma_1} + \beta_2 x_2^{\gamma_2} + ... )$

where $x_0$ is another predictor and $\gamma, \delta$ more coefficients. Both models have a sensible physical interpretation.

  1. Is this a sensible thing to do?

  2. If so, I presume the ridge penalty metaparameter $\lambda$ is replaced by variance on priors for $\beta$. How would I choose the priors?

In the ridge regression $\lambda$ was chosen by generalized cross-validation. In the Bayesian model, is GCV still appropriate for choice of $\lambda$? Is it appropriate for choice of the ordinary parameters $\beta, \gamma, \delta$ as well? If so should I estimate $\beta,\gamma,\delta,\lambda$ together or should I run some nested models where $\lambda$ is deferred to the outer loop?

To be clear, the overarching aim is good out-of-set prediction. I have never used a Bayesian model before so if this is 101 stuff I apologise, please point me at the right resources!

  • 1
    $\begingroup$ It seems to me, that with this change, you are fitting a non linear model with a ridge penalty. That does not make it a Bayesian method, unless you want to put priors on the coefficients. As it stands, you don't need to think of this as a Bayesian method. $\endgroup$ – Greenparker Mar 26 '16 at 11:59
  • 1
    $\begingroup$ Also, are you interested in estimating $\delta^{x_0}$ as a whole or $\delta$ and $x_0$ separately? $\endgroup$ – Greenparker Mar 26 '16 at 12:00
  • $\begingroup$ Taking the second question first, x0 is an independent variable, I'm estimating delta. $\endgroup$ – Sideshow Bob Mar 26 '16 at 23:20
  • $\begingroup$ As to the first question, I'm feeling quite sheepish because I'm not quite sure why I want a Bayesian model! Putting more priors on coefficients does seem like a good way to handle multicollinearity without discarding physically meaningful variables I guess. Also in practical terms I have so far failed to get nonlinear ridge regression working in R. I know a lot of people use Bayesian models for this sort of problem so I wondered if, theory aside, the R packages were just better? ! $\endgroup$ – Sideshow Bob Mar 26 '16 at 23:24
  • $\begingroup$ My attempt at straight nonlinear model: stackoverflow.com/questions/36087868/… $\endgroup$ – Sideshow Bob Mar 26 '16 at 23:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.