# Applying ridge regression on small dataset could improve predicting power?

Suppose I have a small data set $X$ that is $30\times6$. I am wondering if it makes sense to use ridge regression if I want to improve the predicting power of the model.

To my understanding, ridge regression can usually be used to solve the following problems:

1. The data matrix $X$ is singular. In this case, OLS does not work.
2. We have too many features. Ridge regression's objective function puts penalty on size of features, meaning $\sum \|\beta_i\|^2$.

However, I am wondering if the following reasoning makes sense:

OLS estimators are unbiased and have the least variance among all unbiased estimators. Since I have a very small data set, my OLS estimators have very big variance. Even though they are unbiased, since I do not have a big data set, the predicting power might still be low. By using ridge regression, I no longer have unbiased estimators, but a high value of $\lambda$ will give me estimators that have lower variance. As a result, it is possible that I end up with a model that has better predicting power.

• I think "penalty on number of features" more describes LASSO, and is not really true for ridge regression. – GeoMatt22 Sep 7 '16 at 2:40
• With 30 observations and 6 features, I think your best answer is actually applying both methods and testing which does a better job instead of guessing which theory might fit better. Ridge regression helps when there are a large number of multicollinear features which make it difficult to model small datasets. – Arun Jose Sep 7 '16 at 2:44
• @GeoMatt22 Sorry. I meant to say "size of features" – 3x89g2 Sep 7 '16 at 5:20
• It certainly could work. For any given sample size, the optimal amount of $L_2$ penalty is positive (so not zero). That is, there exists a positive $\lambda$ such that ridge will do better than OLS in terms of mean squared error. See Dave Giles blog post "A Regression "Estimator" that Minimizes MSE". – Richard Hardy Sep 7 '16 at 7:20
• @RichardHardy This sounds like a really nice result. Knowing existence is always good. – 3x89g2 Sep 7 '16 at 7:26

Given a correctly specified linear model, the optimal amount of $L_2$ penalty is positive, for any given sample size. That is, there exists a positive $\lambda$ such that ridge will do better than OLS in terms of mean squared error of estimated parameters; see Dave Giles blog post "A Regression "Estimator" that Minimizes MSE" for details. If you are able to estimate $\lambda$ precisely enough, ridge will yield more precise estimates than OLS, which in turn will improve prediction accuracy.