First time posting - will try to be as clear as possible! I'm running a ridge regression, and am having problems calculating the right bias factor (k) to use. I have 4 predictors, one of which is an interaction between 2 predictors. I'd like to use the HKB (Hoerl, Kennard and Baldwin, 1975) method for estimating the the bias factor (I realise there's a lot of discussion about which method is best, but I thought I'd start with this one, as it appeared relatively straightforward). The formula I'm using is:

enter image description here

(...so the number of predictors x the residual mean square, over the sum of the coefficients squared).

When I try to use this formula to calculate k for my ridge regression, I get a huge number (~25). Looking at the ridge trace plots, I was expecting something more in the region of 0.5... (I've posted the ridge trace below - you can see three of the predictors level out early, but one levels out much later. I had assumed that the right k would be somewhere between these points.)

enter image description here

I've been using "Regression Analysis by example " (by Chatterjee and Hadi), which has a walked through example of using this method. A few things I noticed about the example in the book was that their coefficients are generally larger than mine, and their residual mean square is smaller.

My coefficents (Bs) from when k=0 (i.e. a normal OLS regression) are : -.058, -.008, 0.365, and .030.


Residual mean square = 0.89

...so for me: k= 4 x 0.89 / (-0.058)2 + (-.008)2 +(0.365)2 +(.030)2 = 25.85

So my questions are:

  • Is a k of 25 normal, or is it clearly an error? My understanding is that you might sometimes get a k in excess of 1, but 25 seems very high!

  • Does a k this high suggest that a ridge regression is inappropriate? Maybe to do with my small coefficients...?

  • I standardized my variables before running the ridge regression (I'm using the SPSS Ridge Macro), so all the variables have a mean of 0 and SD of 1. My understanding was that variables should be standardized before going into a ridge regression. Could this be the source of my problem?

Any ideas greatly appreciated!

UPDATE: Having double checked my workings using other examples, I'm pretty confident it has nothing to do with standardising the variables. This formula seems to work fine for getting k values for datasets other than mine!! I can therefore only assume my dataset violates the assumptions of a ridge regression in some way.

Could skewed variables give a result like this? Even my standardised variables are not normally distributed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.