4
$\begingroup$

When adjusting a Random Forest or LASSO model to some data, let's say y ~ x1 + x2, do I lose anything by considering transformation of the features to my model? For example: y ~ x1 + x2 + log(x1) + log(x2) + sqrt(x1^2 - x2^2) + x1^2. While it adds computational costs, I feel like it's saying: "I'm not sure the relation between the response and this feature is linear, so let the model choose whether it's linear, or logarithm, or polynomial, etc.

I can see positive points of including the transformed features, but are there negative consequences other than computational costs?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ There is always a risk of overfitting that increases the more features you ask the model to disentangle. In practice, when you're unsure, it's better to explicitly express your ignorance by using a flexible basis expansion like cubic splines instead of rigid options like logarithms, sinusoids, etc. Of course, if you have some domain reason to believe some specific transformation is appropriate, you should include it. $\endgroup$
    – Matthew Drury
    Commented Sep 8, 2018 at 22:02
  • $\begingroup$ Won't overfitted models perform poorly in cross-validation? My first intuition was that if the transformed variables are not really important the penalization (in lasso for example), would exclude them, so my only concern would be computational cost. $\endgroup$
    – VFreguglia
    Commented Sep 8, 2018 at 22:06
  • 2
    $\begingroup$ For sure, LASSO is designed to work that way. But tools can always fail, and more variables does introduce more risk of failure. $\endgroup$
    – Matthew Drury
    Commented Sep 8, 2018 at 22:10
  • 1
    $\begingroup$ You can never guarantee that your algorithms won't make a mistake with irrelevant features added, thereby increasing variance, but OTOH leaving out relevant features increases bias; it's a tradeoff which is hard to make without domain-specific knowledge. Random forests seem to be impacted less by lots of irrelevant features than, say, gradient boosting machines (I've done some rather extensive simulation based upon some large problems we have at work) and LASSO with a kernel is just kernel regression with an $L_1$ penalty; if you've got a lot of data I would guess it would be OK. $\endgroup$
    – jbowman
    Commented Sep 8, 2018 at 22:34

1 Answer 1

Reset to default
3
$\begingroup$

The typical LASSO model is an ordinary linear model plus the a penalty: $$ \min_\beta L(y,g(X\beta))+\lambda \|\beta \|_1 $$

I want to emphasize that the LASSO model is a linear model, so the coefficients are estimating a function that is linear in its parameters. Extensions of a LASSO to GLMs are still linear in the parameters under estimation. In this sense, basis expansions or variable transformations can be powerful tools to improve the expressive power of your model. Likewise, this great power comes the responsibility to avoid overfitting.

In an ideal world, we would know exactly which sets of transformations are the correct ones, and then the modeling task reduces to a linear model. But this is rarely the case, so random forests can do the heavy lifting for us.

The random forest model is an ensemble of many different decision trees. It should be obvious that decision trees are nonlinear: for any binary split, the daughter nodes yield distinct constant functions. The effect of many such binary splits is to divide the feature space into a number of axis-aligned rectangles, each with a different estimate.

Arbitrarily many binary, axis-aligned splits can approximate a complex boundary by using simpler shapes. The classic example is to consider a binary classification task with a perfect linear decision boundary on the line $x_1 + x_2 > c$. This manifests as a diagonal split. Clearly a single axis-aligned split can't approximate a diagonal very well, but many axis aligned splits, you can make a "stair-step" shape that can approximate the diagonal arbitrarily well. Likewise, the same is true for approximating relationships like logarithms or quadratics or sinusoids.

My tangentially-related answer here provides some more elaboration. Can a random forest be used for feature selection in multiple linear regression?

These threads address the random forest piece, but not the LASSO piece.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.