1
$\begingroup$

I am currently examining which of sixteen variables are the most important in predicting a binary outcome. There are 907 observations, so obviously $N$ is much larger than $p$

In the last six months after doing a short course and some reading, I have come to the understanding that older methods for variable selection such as stepwise regression are biased and can lead to incorrect inferences, and that penalised regression techniques are better for such tasks.

However most of the papers and posts on CV that I have read (e.g. here and here) that discuss such techniques (e.g. LASSO and Horseshoe regression) say that they are for high-dimensional data when $p>>n$.

Hence I am starting to wonder if it is appropriate to use regularization techniques when $N>>p$?

In low-dimensional data situations like mine do techniques like penalised regression offer an advantage over standard regression?

$\endgroup$
1
$\begingroup$

Absolutely.

Ridge regression for example can guarantee a reduction in the MSE of $\hat{\beta}$ in non-trivial scenarios with a carefully selected penalty parameter.

The LASSO can also offer parsimony. There may not be a need to include all parameters in the model.

Ridge/Lasso are also useful for helping models generalise when features are strongly correlated - in situations where ordinary leasy-squares estimates would be very large, but effectively cancel each other out. These large estimates can result in larger errors on out of sample data.

$\endgroup$
1
  • $\begingroup$ Thank you. A big load off my mind given how much work I'd already put in running the things. $\endgroup$ – llewmills Jul 28 '20 at 6:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.