Referencing to an example explained in free google machine learning course

Imagine a linear model with 100 input features:

  • 10 are highly informative.
  • 90 are non-informative.

Assume that all features have values between -1 and 1. How we can avoid L2 regularization causing the model to learn a moderate weight for some non-informative features when they happen to be correlated with the label.

In this case, the model incorrectly gives such non-informative features some of the "credit" that should have gone to informative features ultimately leading to misinformed predictions.! that's insidious.

Two questions:

  1. Could anyone suggest method/s circumvent this problem, keeping all the features within the model & not throwing away by picking the features in-out by hand and observing it by doing many iterations with different features? (this hand-engineering method doesn't seem feasible when we have 100 features among which few are actually informative)?

  2. Also, by "informative" or "non-informative", can't we judge this using watching correlation matrix, if yes, sometimes, people use -ve, 0 & +ve correlated features too? then Is "correlation matrix" a good metric for assessing "informative" or "non-informative about the features, if not could anyone suggest some other metrics?


1 Answer 1


A few ideas (out of many possible approaches)

  • Use PCA to reduce the dimensionality (i.e. keep top 10 PCA dimensions)
  • Use some form of feature selection to remove the non-informative features
  • Use a different machine learning model to select top features (e.g. random forest, gradient boosting...)
  • Use lasso regression with an appropriate regularization value
  • Use ElasticNet regularization (which is a combination of l1 and l2 really)
  • ...

... and then perform L2 regression on the resulting features

For your second question: correlation would only work if your features are linearly related to the outcome. Wont work for polynomial or interaction terms or anything else non linear...

  • $\begingroup$ all except PCA make sense to me. If I do PCA, I am gonna completely lose the original features, How I will bring out top 10 dimensions a.k.a features, in my understanding after PCA top 10 features would be a summation of one or the another out of 100 single feature all decided by the PCA algorithm, correct me if I am wrong! $\endgroup$
    – Anu
    Nov 4, 2018 at 4:02
  • $\begingroup$ You are right you will loose the original $\endgroup$ Nov 4, 2018 at 4:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.