What are the 'indications' (i.e. when to use) and 'contra-indications' (i.e. when not to use) of ridge-regression. I tried to read up on the net and it seems to useful when multi-collinearity is there amongst the predictor variables. But what about other situations / assumptions for ridge regression? Do the predictor variables need to be normally distributed? How does it fare with small data sets? Is there a problem if categorical predictor variables are included? How many predictor variables can be included relative to number of rows, etc. Thanks for your insight.
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
The predictor variables don't need to be normally distributed. Instead, the noise term $\epsilon$ is normally distributed: $$y=w^{T}x+\epsilon$$ If we build the model using maximum likelihood estimates, then it's just linear regression without regularization. If we assume the weights have a Gaussian prior and use MAP estimates, then it will become ridge regression. But in either case, predictor variables need not be normally distributed.
As for when to use ridge regression, in most cases ridge regression outperforms other regularization methods (say L1-norm). You can refer to the following FAQ page by the LIBSVM author:
https://www.csie.ntu.edu.tw/~cjlin/liblinear/FAQ.html#l1_regularized_classification
Therefore I will say try ridge regression first. You may also want to try L1-regularization (LASSO) or elastic nets when:
You know some of the features you are including in your model might be zero (i.e., you know the some coefficients in the "true model" are zero)
Your features do not highly correlate with each other
You want to perform feature selection but don't want to use wrapper/filter approaches
The above reasons are exactly the nature of L1-regularization. The LASSO yields sparse output, which can be viewed as built-in feature selection. However it only selects one feature from a bunch of highly correlated features.
$\begingroup$
$\endgroup$
I refer to this pdf
In formula (3.6) you find the formula for the estimator of $\hat{\beta}$. This formula learns that one has to invert the matrix $(X^TX)$, so you may have problems with Least squares if this matrix is (almost) not invertible. This is the case of there is multicollinearity between the independent variables (the $X$) or if you have more independent variables than observations.
For ridge regression you have to use formula (3.44) so you have to invert $(X^T X+\lambda I)$. because of the presence of $\lambda$, this matrix is invertible whenever $\lambda >0$. So for $\lambda > 0$ the matrix is invertible, also in case of (1) multicollinearity and (2) in the case where you have more independent variables than observations.
These are the two cases where ridge has an advantage over OLS.
Ridge regression needs the same assumptions or has the same properties as OLS. Indeed, most of OLS's properties stemm from the fact that $\hat{\beta}^{(ols)}$ is a linear combination of the $y$, as can be seen from the formula (3.4). The formula (3.44) shows that for ridge regression the $\hat{\beta}^{(ridge)}$ is also a linear combination of the $y$'s, and consequantly all properties of the OLS-estimators that were derived from this linearity will also hold for ridge regression.