Suppose I have a small data set $X$ that is $30\times6$. I am wondering if it makes sense to use ridge regression if I want to improve the predicting power of the model.
To my understanding, ridge regression can usually be used to solve the following problems:
- The data matrix $X$ is singular. In this case, OLS does not work.
- We have too many features. Ridge regression's objective function puts penalty on size of features, meaning $\sum \|\beta_i\|^2$.
However, I am wondering if the following reasoning makes sense:
OLS estimators are unbiased and have the least variance among all unbiased estimators. Since I have a very small data set, my OLS estimators have very big variance. Even though they are unbiased, since I do not have a big data set, the predicting power might still be low. By using ridge regression, I no longer have unbiased estimators, but a high value of $\lambda$ will give me estimators that have lower variance. As a result, it is possible that I end up with a model that has better predicting power.