If there any benefit to using ridge regression in a simple linear regression problem where the aim is prediction?

Question

Consider the following situation:

• We have a simple linear regression model (as opposed to a multiple regression model or a polynomial regression model).
• We are interested in prediction rather than inference.
• We have far more observations than predictors.

In this situation is there any benefit to using ridge regression? From what I understand ridge regression allows us to reduce overfitting in complex models at the cost of introducing some bias.

But in the above situation we don't have a complex model; in particular since we are using simple linear regression rather than polynomial regression we are modelling the relationship with a straight line as opposed to a curve.

So can ridge regression still provide us with any benefits in this scenario? Can we still obtain better predictions, by using ridge regression to reduce variance at the cost of introducing bias, when the model is so simple?

Answer 1

Ridge allows you to regularize ("shrink") coefficient estimates made by linear regression (OLS). This means that the estimated coefficients are pushed towards 0, to make them work better on new data-sets ("optimized for prediction"). This allows you to use complex models and avoid over-fitting at the same time. Despite OLS being the best linear unbiased estimator, ridge can demonstrably achieve a lower MSE than OLS by being a biased estimator. Now let's discuss the non-complex (linear problem) case,

In machine learning, if we want to make predictions, the attention is not limited to the model itself, but the unseen data as well (the test data). Sure, the model and training data might suggest that everything complies with linear regression and that we don't have to resort to ridge or lasso, but this doesn't stop the potential of the new unseen data from being significantly different than what you saw in-sample, so ridge would still be of some benefit, instead of OLS, in regards to what you don't possess upon training: the test data. Since the test data is what you make predictions from, not the training data.

Ridge is most useful when there is multicollinearity in the features however, since its main purpose is to treat multicollinearity in the features. Since I used the word features, rather than training data, it means that the features upon training might be simple enough and non-complex, but might have very different relationships within the unseen test data. So, again, even we don't observe collinearity in-sample, there could very well be multicollinearity arising in the unseen test data, so ridge would be the best safeguard of the regularization methods in this scenario.

but regardless of scenario, as long as the task is prediction, ridge will actively work to reduce overfitting compared to OLS as can be seen in ridge's objective function. The better question is: what is the optimal $\lambda^*$ for a given prediction problem, and how different is it from $\lambda=0$, which is where ridge collapses to OLS? If you use cross-validation and find out $\lambda^*$ is far from $\lambda=0$ (the definition of "far" might be tricky for potentially very tiny values for $\lambda$, which depends on the units and scale of the data itself), then it will show ridge paid off. Otherwise, if $\lambda^*\approx 0$, then ridge might've only provided a small benefit over OLS.