0
$\begingroup$

I heard people ask which one is better: Linear regression with regularization or Random Forest. My question is why can't you use regularization with Random Forest?

My understanding is that different regularization technique is adding a term to cost functions such as cross-entropy to reduce accuracy/overfitting to training data.

Typically, preventing overfitting technique for decision trees is associated with using Random Forest. I have never heard people associate regularization with decision trees or Random Forests.

Is it that reducing overfitting at the individual node level is too complex?

Edit: I acknowledge that RF is already a regularization method in a sense for decision trees through bootstrapping and aggregating. I think RF can be imaged as trying to estimate the answer from aggregating answers from crowd versus an expert(aka wisdom of the crowd). I am not asking what regularization for RF is, I am asking why can't we simply also apply regulation at a single node step of the decision tree, since it is just adding bias to the cross-entropy, which is the cost function for linear regression or the decision used to split a tree. Is it because there is no point doing that since we don't use cross-entropy to update values of the feature?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
3
$\begingroup$

Random forest has regularization, it's just not in the form of a penalty to the cost function. Random forest doesn't have a global cost function in the same sense of linear regression; it's just greedily maximizing information gain at each split. Limiting child node size, minimum information gain and so on all change how the trees are constructed and impose regularization on the model in the sense that a proposed split must be "large enough".

Improve this answer
$\endgroup$
6
  • $\begingroup$ My understanding is that Random forest itself is a regularization method for decision trees by repeating various forms of decisions trees and have some fitting better to some data and other fitting better to others. My question was why don't we just apply regularization at each node of a single decision tree.n $\endgroup$
    – LindseyPeng
    Jul 17, 2019 at 15:43
  • 1
    $\begingroup$ Ordinary decision trees have low bias but high variance. Averaging many randomized trees, a.k.a random forest, creates a low-bias, low-variance estimator. This is a kind of regularization on its own. However, the "default" RF method does not regularize how splits are formed (instead making purely greedy splits). This is where child node size and minimum information gain enter the picture: change how the splits are formed to be less sensitive to "too small" effects. $\endgroup$
    – Sycorax ♦
    Jul 17, 2019 at 15:45
  • $\begingroup$ I ddon't think the proposed split to be 'large enough' is regularization ?. I believe that proposing each split must be large enough is reducing bias . I imagine decision tree as drawing these boundaries in n-dimensional space that accurately classify samples. Random forest is a regularization technique itself with boostrapping and aggregating results of decision trees. $\endgroup$
    – LindseyPeng
    Jul 17, 2019 at 15:46
  • $\begingroup$ also I feel like even though RF has its own regularization technique dosn't explain my question , which is already acknowledging that RF is often associated with regularization on its own. My question is why can't we also just apply regularization(like adding a bias term to cross-entropy (which is also sometimes used for splitting a node)) at each decision step?) $\endgroup$
    – LindseyPeng
    Jul 17, 2019 at 15:49
  • 1
    $\begingroup$ (1) Can you write down an expression for what that regularization would look like? Random forest doesn't have a cost function in the same sense of linear regression, so it's not obvious what connection you're drawing between the two. (2) Imposing a constraint on the class of models that you can estimate is pretty much the definition of increasing bias. Think about the equivalence between ridge regression penalizing the norm of the coefficient vector and imposing a constraint that the norm of the coefficient vector must be smaller than some constant. $\endgroup$
    – Sycorax ♦
    Jul 17, 2019 at 15:51
1
$\begingroup$

In fact, there are some works related to using regularization on decision tree splitting. For example, in [1], the author proposes a cost function to penalize redundant features, and thus get a more succinct feature set. The experimental results reveal that it can effectively get a more concise feature subset without loss of accuracy.

[1]. Deng H, Runger G. Feature selection via regularized trees[C]//The 2012 International Joint Conference on Neural Networks (IJCNN). IEEE, 2012: 1-8.

Improve this answer
$\endgroup$
0
$\begingroup$

Is it because we can't update the value of feature that is used to split each nodde? Decision Tree uses features to split data set, and sometimes there is no continuous value associated with that feature. Regularization is applied to a cost function that is used to update the values of parameters. Since we are not trying to predict the correct value for the feature in Decision trees then we don't prevent overfitting by changing the value of the feature.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.