1
$\begingroup$

First time posting - will try to be as clear as possible! I'm running a ridge regression, and am having problems calculating the right bias factor (k) to use. I have 4 predictors, one of which is an interaction between 2 predictors. I'd like to use the HKB (Hoerl, Kennard and Baldwin, 1975) method for estimating the the bias factor (I realise there's a lot of discussion about which method is best, but I thought I'd start with this one, as it appeared relatively straightforward). The formula I'm using is:

enter image description here

(...so the number of predictors x the residual mean square, over the sum of the coefficients squared).

When I try to use this formula to calculate k for my ridge regression, I get a huge number (~25). Looking at the ridge trace plots, I was expecting something more in the region of 0.5... (I've posted the ridge trace below - you can see three of the predictors level out early, but one levels out much later. I had assumed that the right k would be somewhere between these points.)

enter image description here

I've been using "Regression Analysis by example " (by Chatterjee and Hadi), which has a walked through example of using this method. A few things I noticed about the example in the book was that their coefficients are generally larger than mine, and their residual mean square is smaller.

My coefficents (Bs) from when k=0 (i.e. a normal OLS regression) are : -.058, -.008, 0.365, and .030.

p=4

Residual mean square = 0.89

...so for me: k= 4 x 0.89 / (-0.058)2 + (-.008)2 +(0.365)2 +(.030)2 = 25.85

So my questions are:

  • Is a k of 25 normal, or is it clearly an error? My understanding is that you might sometimes get a k in excess of 1, but 25 seems very high!

  • Does a k this high suggest that a ridge regression is inappropriate? Maybe to do with my small coefficients...?

  • I standardized my variables before running the ridge regression (I'm using the SPSS Ridge Macro), so all the variables have a mean of 0 and SD of 1. My understanding was that variables should be standardized before going into a ridge regression. Could this be the source of my problem?

Any ideas greatly appreciated!


UPDATE: Having double checked my workings using other examples, I'm pretty confident it has nothing to do with standardising the variables. This formula seems to work fine for getting k values for datasets other than mine!! I can therefore only assume my dataset violates the assumptions of a ridge regression in some way.

Could skewed variables give a result like this? Even my standardised variables are not normally distributed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.