Lets say we have a dataset with hundreds of features. Since I'm not really sure whether all these features are good identifiers or not, I think we might need use one or all of the following techniques in the preprocessing step (before feed the data into a classifier):

  • Use VIF to test and drop those multi-collinear features
  • Use PCA, LDA etc. to create uncorrelated features
  • Use function as SelectFromModel in scikit-learn package to find the most important features
  • Standardise

However, when I tried to apply them in a particular problem, I'm a kind of lost,

  1. In what situation can we make sure that the preprocessing step is enough?
  2. I think the four step I mentioned above deal with different problems that may happens in the data, so I just wondering if it's appropriate I always concatenate all of them in the preprocessing step?

Standardizing is a technical trick. It speeds up convergence. It doesn't change anything fundamentally in a reasonable model. Think of using 100mm, 10cm or 1m as a length of the box. Would a model depend on the unit of measure? A sensible model will not.

PCA is used to get orthogonal factors, but not in the context of machine learning, usually. In ML it's most common application is dimensionality reduction. It's a reasonable approach to highly correlated data, but in many cases autoencoder could be preferable. So, it's not a must do step.

I never used VIF even in regression. Multi collinearity is not a big issue in machine learning. In severe cases where you have loads of almost identical variables, it can help with performance to remove collinear variables.

If you're learning ML I wouldn't bother too much about feature engineering in the beginning. Just feed all variables into the model and see what happens.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.