How we can avoid making L2 regularization causing the model to learn a moderate weight for some non-informative features.?

Question

Referencing to an example explained in free google machine learning course

Imagine a linear model with 100 input features:

• 10 are highly informative.
• 90 are non-informative.

Assume that all features have values between -1 and 1. How we can avoid L2 regularization causing the model to learn a moderate weight for some non-informative features when they happen to be correlated with the label.

In this case, the model incorrectly gives such non-informative features some of the "credit" that should have gone to informative features ultimately leading to misinformed predictions.! that's insidious.

Two questions:

1. Could anyone suggest method/s circumvent this problem, keeping all the features within the model & not throwing away by picking the features in-out by hand and observing it by doing many iterations with different features? (this hand-engineering method doesn't seem feasible when we have 100 features among which few are actually informative)?

2. Also, by "informative" or "non-informative", can't we judge this using watching correlation matrix, if yes, sometimes, people use -ve, 0 & +ve correlated features too? then Is "correlation matrix" a good metric for assessing "informative" or "non-informative about the features, if not could anyone suggest some other metrics?

Answer 1

A few ideas (out of many possible approaches)

• Use PCA to reduce the dimensionality (i.e. keep top 10 PCA dimensions)
• Use some form of feature selection to remove the non-informative features
• Use a different machine learning model to select top features (e.g. random forest, gradient boosting...)
• Use lasso regression with an appropriate regularization value
• Use ElasticNet regularization (which is a combination of l1 and l2 really)
• ...

... and then perform L2 regression on the resulting features

For your second question: correlation would only work if your features are linearly related to the outcome. Wont work for polynomial or interaction terms or anything else non linear...