25
$\begingroup$

Machine learning (ML) uses linear and logistic regression techniques heavily. It also relies on feature engineering techniques (feature transform, kernel , etc).

Why is nothing about variable transformation (e.g.power transformation) mentioned in ML? (For example, I never hear about taking root or log to features, they usually just use polynomials or RBFs.) Likewise, why don't ML experts care about feature transformations for the dependent variable? (For example, I never hear about taking the log transformation of y; they just don't transform y.)

Edits: Maybe the question is not definitely, my really question is "is power transformation to variables not important in ML?"

Improve this question
$\endgroup$
2
  • 4
    $\begingroup$ I'd like to know why this was downvoted; it's actually an interesting question. $\endgroup$
    – shadowtalker
    Commented Jan 10, 2015 at 15:37
  • 1
    $\begingroup$ I think most people would have taken a linear regression course prior to their first ML course. Surely, the stock LR course would contain a chapter on these things (transformations). Btw, I didn't downvote the question. $\endgroup$
    – user603
    Commented Jan 10, 2015 at 18:02

3 Answers 3

Reset to default
13
$\begingroup$

The book Applied Predictive Modeling by Kuhn and Johnson is a highly-regarded practical machine learning book with a large section on variable transformation including Box-Cox. The authors claim that many machine learning algorithms work better if the features have symmetric and unimodal distributions. Transforming the features like this is an important part of "feature engineering".

Improve this answer
$\endgroup$
8
$\begingroup$

Well from my own perspective, quite often I am interested in the predictive distribution of the response variable, rather than just the conditional mean, and in that case it is better to use a likelihood that more correctly represents the target distribution. For instance, I like to use kernelised linear models rather than (say) support vector regression, because I can use a Poisson likelihood if I want to. As a lot of machine learning people are Bayesians, I suspect that using a different likelihood will seem more elegant than transformations (choosing an appropriate likelihood is generally the first step).

Improve this answer
$\endgroup$
0
$\begingroup$

Here is my afterward thoughts.

I think it's because ML is largely deal with classification, and classification is no need for transforming y (y is categorical). ML usually deal with large independent variables (e.g. thousands in NLP ) and logistic regression doesn't require normality; I think that's why they don't use Box-Cox power transformation due to speed consideration. (note: I'm not familiar to power transformation.)

Improve this answer
$\endgroup$
1
  • $\begingroup$ I thought that this answer to your own question seemed odd, until I reread your post and saw that it was about y, not x. So, yes, for classification, I would agree that transforming y does not make much sense. Transforming x can still help a lot of classification methods, from logistic regression to deep learning. $\endgroup$
    – Jared Becksfort
    Commented Jan 20, 2022 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.