2
$\begingroup$

I'm a newbie to machine learning. I'm working on a dataset to predict a target variable in terms of the independent variables. In the dataset the independent variables few are very highly correlated; correlation coefficient is 1.

I used ridge regression since it will take care of the correlated variables.

The algorithm is penalizing all the correlated variables and taking only one among them.

Can someone throw light on how the variable selection is done for ridge regression in presence of correlated variables?

$\endgroup$
  • 2
    $\begingroup$ Yes, ridge regression will take care of the correlated variables. But no, ridge regression does not do variable selection. That is the benefit of ridge regression: why throw away some variables if they contribute similar information (because they are highly correlated); better use all of them. $\endgroup$ – Richard Hardy May 16 '17 at 11:36
  • $\begingroup$ @RichardHardy but from my model only one among the correlated variables are selected. Is there any rational for that? $\endgroup$ – naveen marri May 16 '17 at 11:37
  • 3
    $\begingroup$ That means you are using something else than ridge regression. $\endgroup$ – Richard Hardy May 16 '17 at 11:38
1
$\begingroup$

As mentioned in the Richard Hardy's comment, ridge regression will NOT select variable but using all of them. The statement

The algorithm is penalizing all the correlated variables and taking only one among them.

is wrong.


Here is an example to show ridge regression will use all variables.

We build a toy mtcars data that has $2$ identical columns on a car's weight.

Note the output of the ridge regression that has coefficients on both wt and wt2.

> d=mtcars[,c('mpg','wt')]
> d$wt2=d$wt
> head(d)
                   mpg    wt   wt2
Mazda RX4         21.0 2.620 2.620
Mazda RX4 Wag     21.0 2.875 2.875
Datsun 710        22.8 2.320 2.320
Hornet 4 Drive    21.4 3.215 3.215
Hornet Sportabout 18.7 3.440 3.440
Valiant           18.1 3.460 3.460

> library(MASS)
> ridge <- lm.ridge (mpg ~ wt+wt2,d, lambda = 0.1)
> ridge
                 wt       wt2 
37.258302 -2.668067 -2.668067 

Mathematically, the ridge regression is adding L2 regularization on objective, which is

$$ \text{minimize}~~ (\| X\beta-y\|^2+\lambda\|\beta\|^2) $$

Here $X$ is the design matrix, $\beta$ is a vector of coefficients and $y$ is a vector of response variable.

Suppose data matrix $X$ has two identical columns, i.e., correlation $1$ on two features, Adding the regularization will make matrix $X^TX$ non-singular and still able to solve for the $\beta$.

In summary, here is how ridge regression handle the collinearity problem comparing to OLS:

  • OLS: $\beta$ can be obtained by solving the linear system $X^TX\beta=X^Ty$
  • Ridge: $\beta$ can be obtained by solving the linear system $(X^TX+\lambda I)\beta=X^Ty$, where $I$ is Identity matrix.
$\endgroup$
1
$\begingroup$

The commenters and the answer from @hxd1011 are certainly correct when there is high but not perfect correlation among predictors. In that case ridge regression will include all predictors. But you seem to have perfect correlation among some predictors, which adds an additional complication.

If two predictor variables have a correlation coefficient of exactly 1, then they are exactly linearly dependent. Then the rank of the design matrix is less than the number of predictors. For standard linear regression, depending on the software being used this will either lead to an error or to a (perhaps hidden) choice of one variable by the algorithm.

It seems that this is what is occurring with your attempt at ridge regression; your software might be simply throwing out all but one of a set of variables that are exactly linearly dependent. In that case of perfect linear correlation, it doesn't really matter which one is chosen as the variables are all equivalent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.