I have just started machine learning and was asked this concept-based question,

"Suppose you are working on a stock market prediction model and the data you collected have millions of features, what should you do?"

I found two possible methods - Regularization and dimensionality reduction. But I was told that regularization is incorrect because it does not affect input data but only the output data. Whereas dimensionality reduction removes unnecessary/useless data that generates noise.

My main question is, if excessive features in a dataset could cause overfitting and regularization can help to reduce the complexity of the model, why is regularization not a valid solution?

Would sincerely appreciate if anyone could provide some usage examples for both methods.


Interesting question - I assume that you want to build a classifier that predicts for example whether the price of a stock goes up - or down - during the coming day.

My advice is to take a different path. With literally millions of feature variables available, I would start off with a machine learning method that uses feature selection as an integral part of its building process. I would go for C4.5 or other decision tree classifiers which perform sequential forward feature search during learning.

You will end up with a classifier model with a small subset of well-predicting features. Use that subset of features to train other classifiers for comparison, neural networks, discriminant analysis, logistic regression - a support vector machine.

Regularization is mainly applied for reducing model complexity - a different purpose than feature selection. Dimensionality reduction is applicable - but most often comes down to principal component analysis (PCA) - a technique which assumes normally distributed data.

  • $\begingroup$ Thank you for your answer! Am I right to say that when one collects a set of data, the first step will be feature selection (Determining the useful features) and that regularization will only be used at a later stage if the model that was trained is not generalized enough? $\endgroup$ – Wb16 Aug 18 '20 at 2:19
  • $\begingroup$ Yes - that would be the correnct order in which to do things. In your case, you have a very lager number of features ('millions') which is way too much for algorithms to handle. So a forward feature selection approach is needed before you look into model complexity. $\endgroup$ – Match Maker EE Aug 18 '20 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.