2
$\begingroup$

I have just started machine learning and was asked this concept-based question,

"Suppose you are working on a stock market prediction model and the data you collected have millions of features, what should you do?"

I found two possible methods - Regularization and dimensionality reduction. But I was told that regularization is incorrect because it does not affect input data but only the output data. Whereas dimensionality reduction removes unnecessary/useless data that generates noise.

My main question is, if excessive features in a dataset could cause overfitting and regularization can help to reduce the complexity of the model, why is regularization not a valid solution?

Would sincerely appreciate if anyone could provide some usage examples for both methods.

Improve this question
$\endgroup$
1
  • $\begingroup$ But I'm wondering if you use a tree classifier in Features Selection, you can get a Best subset, but if you use another classifier, will the Best subset still be the same? $\endgroup$
    – Ehu Li
    Oct 15, 2021 at 4:20

1 Answer 1

Reset to default
2
$\begingroup$

Interesting question - I assume that you want to build a classifier that predicts for example whether the price of a stock goes up - or down - during the coming day.

My advice is to take a different path. With literally millions of feature variables available, I would start off with a machine learning method that uses feature selection as an integral part of its building process. I would go for C4.5 or other decision tree classifiers which perform sequential forward feature search during learning.

You will end up with a classifier model with a small subset of well-predicting features. Use that subset of features to train other classifiers for comparison, neural networks, discriminant analysis, logistic regression - a support vector machine.

Regularization is mainly applied for reducing model complexity - a different purpose than feature selection. Dimensionality reduction is applicable - but most often comes down to principal component analysis (PCA) - a technique which assumes normally distributed data.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your answer! Am I right to say that when one collects a set of data, the first step will be feature selection (Determining the useful features) and that regularization will only be used at a later stage if the model that was trained is not generalized enough? $\endgroup$
    – Wb10
    Aug 18, 2020 at 2:19
  • $\begingroup$ Yes - that would be the correnct order in which to do things. In your case, you have a very lager number of features ('millions') which is way too much for algorithms to handle. So a forward feature selection approach is needed before you look into model complexity. $\endgroup$
    – Match Maker EE
    Aug 18, 2020 at 7:18
  • $\begingroup$ My understanding is that PCA does not assume normality - it is adjuvant statistical analysis that generally makes that assumption. PCA is a mathematical transformation, not a statistical tool. See stats.stackexchange.com/questions/200410/… $\endgroup$
    – ReneBt
    Oct 15, 2021 at 5:30
  • $\begingroup$ Feature selection is not necessarily the best approach. You have no guarantee that feature selection algorithm would choose the features that are the most useful for the ML algorithm. If you use different feature selection algorithms, they are likely to select different features. $\endgroup$
    – Tim ♦
    Oct 15, 2021 at 5:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.