Skip to main content

Questions tagged [cart]

'Classification And Regression Trees', also sometimes called 'decision trees'. CART is a popular machine learning technique, and it forms the basis for techniques like random forests and common implementations of gradient boosting machines.

Filter by
Sorted by
Tagged with
141 votes
9 answers
59k views

Obtaining knowledge from a random forest

Random forests are considered to be black boxes, but recently I was thinking what knowledge can be obtained from a random forest? The most obvious thing is the importance of the variables, in the ...
Tomek Tarczynski's user avatar
65 votes
5 answers
75k views

Training a decision tree against unbalanced data

I'm new to data mining and I'm trying to train a decision tree against a data set which is highly unbalanced. However, I'm having problems with poor predictive accuracy. The data consists of students ...
chrisb's user avatar
  • 915
52 votes
3 answers
72k views

What is Deviance? (specifically in CART/rpart)

What is "Deviance," how is it calculated, and what are its uses in different fields in statistics? In particular, I'm personally interested in its uses in CART (and its implementation in rpart in R). ...
Tal Galili's user avatar
  • 21.7k
43 votes
1 answer
51k views

Relative variable importance for Boosting

I'm looking for an explanation of how relative variable importance is computed in Gradient Boosted Trees that is not overly general/simplistic like: The measures are based on the number of times a ...
Antoine's user avatar
  • 6,179
6 votes
1 answer
3k views

What is the test statistics used for a conditional inference regression tree?

In Hothorn et al, the test statistic is specified as $$ T_j(L_n, w) = vec(\sum w_i g_j(X_{ji}) h(Y_i, (Y_1,...,Y_n)^T))$$ What is the exact form of this test statistic with a continuous response and ...
goldisfine's user avatar
73 votes
3 answers
172k views

How to actually plot a sample tree from randomForest::getTree()? [closed]

Anyone got library or code suggestions on how to actually plot a couple of sample trees from: getTree(rfobj, k, labelVar=TRUE) (Yes I know you're not supposed ...
smci's user avatar
  • 1,496
19 votes
2 answers
17k views

What is the VC dimension of a decision tree?

What is the VC dimension of a decision tree with k splits in two dimensions? Let us say the model is CART and the only allowed splits are parallel to the axes. So for one split we can order 3 points ...
Tal Galili's user avatar
  • 21.7k
48 votes
3 answers
38k views

How are Random Forests not sensitive to outliers?

I've read in a few sources, including this one, that Random Forests are not sensitive to outliers (in the way that Logistic Regression and other ML methods are, for example). However, two pieces of ...
makansij's user avatar
  • 2,289
37 votes
1 answer
74k views

How do decision tree learning algorithms deal with missing values (under the hood)

What are the methods that decision tree learning algorithms use to deal with missing values? Do they simply full the slot in using a value called missing?
user1172468's user avatar
  • 2,035
16 votes
2 answers
15k views

Mathematics behind classification and regression trees

Can anyone help explain some of the mathematics behind classification in CART? I'm looking to understand how two main stages happen. For instance I trained a CART classifier on a dataset and used a ...
G Gr's user avatar
  • 1,021
169 votes
3 answers
181k views

Gradient Boosting Tree vs Random Forest

Gradient tree boosting as proposed by Friedman uses decision trees as base learners. I'm wondering if we should make the base decision tree as complex as possible (fully grown) or simpler? Is there ...
FihopZz's user avatar
  • 2,013
24 votes
3 answers
8k views

What is the relationship between the GINI score and the log-likelihood ratio

I am studying classification and regression trees, and one of the measures for the split location is the GINI score. Now I am used to determining best split location when the log of the likelihood ...
EngrStudent's user avatar
  • 9,430
23 votes
3 answers
5k views

Is a decision stump a linear model?

Decision stump is a decision tree with only one split. It can also be written as a piecewise function. For example, assume $x$ is a vector, and $x_1$ is the first component of $x$, in regression ...
Haitao Du's user avatar
  • 37k
16 votes
2 answers
7k views

Decision Trees and Regression - Can predicted values be outside range of training data?

When it comes to decision trees, can the predicted value lay outside of the range of the training data? For example, if the training data set range of the target variable is 0-100, when I generate my ...
user3788557's user avatar
  • 1,629
12 votes
6 answers
11k views

Making a single decision tree from a random forest

I am using scikit learn to build a Random Forest classifier. I have heard that it might be possible to build a single decision tree from a Random Forest. The suggestion is that although the ...
Simd's user avatar
  • 2,029
37 votes
4 answers
49k views

What is the weak side of decision trees?

Decision trees seems to be a very understandable machine learning method. Once created it can be easily inspected by a human which is a great advantage in some applications. What are the practical ...
Łukasz Lew's user avatar
  • 1,412
30 votes
5 answers
69k views

How to measure/rank "variable importance" when using CART? (specifically using {rpart} from R)

When building a CART model (specifically classification tree) using rpart (in R), it is often interesting to know what is the importance of the various variables introduced to the model. Thus, my ...
Tal Galili's user avatar
  • 21.7k
21 votes
3 answers
16k views

Regression tree algorithm with linear regression models in each leaf

Short version: I'm looking for an R package that can build decision trees whereas each leaf in the decision tree is a full Linear Regression model. AFAIK, the library ...
Jonas Sourlier's user avatar
16 votes
3 answers
21k views

Feature Importance for Linear Regression

Is there a way to find feature importance of linear regression similar to tree algorithms, or even some parameter which is indicative? I am aware that the coefficients don't necessarily give us the ...
Kunal's user avatar
  • 333
13 votes
2 answers
19k views

Will decision trees perform splitting of nodes by converting categorical values to numerical in practice?

In Decision Trees, when doing classification or regression, do we use only numerical values? Suppose I have a categorical column Wind as a feature. Suppose I am ...
Sarath R Nair's user avatar
32 votes
1 answer
72k views

What is "feature space"?

What is the definition of "feature space"? For example, When reading about SVMs, I read about "mapping to feature space". When reading about CART, I read about "partitioning to feature space". I ...
power's user avatar
  • 1,732
25 votes
5 answers
13k views

Alternatives to classification trees, with better predictive (e.g: CV) performance?

I am looking for an alternative to Classification Trees which might yield better predictive power. The data I am dealing with has factors for both the explanatory and the explained variables. I ...
Tal Galili's user avatar
  • 21.7k
17 votes
1 answer
22k views

Gini decrease and Gini impurity of children nodes

I'm working on the Gini feature importance measure for random forest. Therefore, I need to calculate the Gini decrease in node impurity. Here is the way I do so, which leads to a conflict with the ...
Remi Mélisson's user avatar
17 votes
1 answer
16k views

How should decision tree splits be implemented when predicting continuous variables?

I'm actually writing an implementation of Random Forests but I believe the question is specific to decision trees (independent of RFs). So the context is that I'm creating a node in a decision tree ...
redcalx's user avatar
  • 868
16 votes
2 answers
15k views

A simple & clear explanation of the Gini impurity?

In a context of decision tree splitting, it is not obvious to see why the Gini impurity $$ i(t)=1-\sum\limits_{j=1}^k p^2(j|t) $$ is a measure of node t impurity. Is there an easy explanation of this?
Picaud Vincent's user avatar
13 votes
2 answers
7k views

Do Random Forests exhibit prediction bias?

I think this is a straightforward question, although the reasoning behind why or why not may not be. The reason I ask is that I have recently written my own implementation of a RF and although it ...
redcalx's user avatar
  • 868
12 votes
1 answer
3k views

Difference in implementation of binary splits in decision trees

I am curious about the practical implementation of a binary split in a decision tree - as it relates to levels of a categorical predictor $X{j}$. Specifically, I often will utilize some sort of ...
B_Miner's user avatar
  • 8,780
9 votes
1 answer
3k views

CART: Selection of best predictor for splitting when gains in impurity decrease are equal?

My question deals with Classification trees. Consider the following example from the Iris data set: I want to manually select the best predictor for the first split. According to the CART algorithm, ...
Antoine's user avatar
  • 6,179
8 votes
1 answer
4k views

Pooling levels of categorical variables for regression trees

I have a data set I would like to do a regression analysis for. There are many features of both categorical and continuous types. One of the categorical features has many (>75) levels so this is an ...
Keith's user avatar
  • 521
5 votes
1 answer
2k views

Non-perpendicular hyperplane decision trees

The usual implementation of decision trees, and estimators based on ensembles of decision trees, use decision trees that threshold on a single predictor at each split. So they divide the feature space ...
sambajetson's user avatar
2 votes
1 answer
799 views

Test statistics used for a conditional inference regression tree?

Following the question asked previously about the interpretation of the Test Statistic used for Conditional Inference Trees (What is the test statistics used for a conditional inference regression ...
Ismael's user avatar
  • 33
109 votes
1 answer
70k views

Conditional inference trees vs traditional decision trees

Can anyone explain the primary differences between conditional inference trees (ctree from party package in R) compared to the ...
B_Miner's user avatar
  • 8,780
76 votes
2 answers
46k views

Practical questions on tuning Random Forests

My questions are about Random Forests. The concept of this beautiful classifier is clear to me, but still there are a lot of practical usage questions. Unfortunately, I failed to find any practical ...
lithuak's user avatar
  • 1,013
21 votes
1 answer
2k views

Is the sum of two decision trees equivalent to a single decision tree?

Suppose we have two regression trees (tree A and tree B) that map input $x \in \mathbb{R}^d$ to output $\hat{y} \in \mathbb{R}$. Let $\hat{y} = f_A(x)$ for tree A and $f_B(x)$ for tree B. Each tree ...
user20160's user avatar
  • 32.7k
18 votes
4 answers
20k views

why boosting method is sensitive to outliers

I found many articles that state that boosting methods are sensitive to outliers, but no article explaining why. In my experience outliers are bad for any machine learning algorithm, but why are ...
lserlohn's user avatar
  • 347
16 votes
2 answers
7k views

Do CART trees capture interactions among predictors?

This paper claims that in CART, because a binary split is performed on a single covariate at each step, all splits are orthogonal and therefore interactions among covariates are not considered. ...
Antoine's user avatar
  • 6,179
16 votes
2 answers
20k views

How to choose $\alpha$ in cost-complexity pruning?

In the following lectures Tree Methods, they describe a tree algorithm for cost complexity pruning on page 21. It says we apply cost complexity pruning to the large tree in order to obtain a sequence ...
itzjustricky's user avatar
14 votes
3 answers
2k views

Can CART models be made robust?

A colleague in my office said to me today "Tree models aren't good because they get caught by extreme observations". A search here resulted in this thread that basically supports the claim. Which ...
Tal Galili's user avatar
  • 21.7k
13 votes
4 answers
8k views

Can Tree-based regression perform worse than plain linear regression?

Hi I'm studying regression techniques. My data has 15 features and 60 million examples (regression task). When I tried many known regression techniques (gradient boosted tree, Decision tree ...
amityaffliction's user avatar
13 votes
2 answers
23k views

Best practices for coding categorical features for Decision Trees?

When coding categorical features for linear regression, there is a rule: number of dummies should be one less than the total number of levels (to avoid collinearity). Does there exist a similar rule ...
Sergey Bushmanov's user avatar
12 votes
1 answer
2k views

Advantage of GLMs in terminal nodes of a regression tree?

So I'm playing around with the idea of writing an algorithm that grows and prunes a regression tree from the data and then, in the terminal nodes of the tree, fits a GLM. I've been trying to read up ...
ApeWithPants's user avatar
12 votes
3 answers
15k views

How can adding a feature reduce a model's performance?

Context: I'm building a model to predict the type of offense (7 classes) from NYPD data. ...
David Stevens's user avatar
11 votes
1 answer
20k views

Decision trees variable (feature) scaling and variable (feature) normalization (tuning) required in which implementations?

In many machine learning algorithms, feature scaling (aka variable scaling, normalization) is a common prepocessing step Wikipedia - Feature Scaling -- this question was close Question#41704 - How and ...
JasonAizkalns's user avatar
10 votes
3 answers
11k views

How does xgboost select which feature to split on?

Pretty straightforward question. Does it check every feature and pick the one that maximizes some split-quality metric (like a decision tree), or does it just select a feature at random (like a random ...
jon_simon's user avatar
  • 2,049
10 votes
1 answer
11k views

Why do Decision Trees/rpart prefer to choose continuous over categorical variables?

I run some decision trees in rpart with 10 continuous variables and 3 categorical variables (with 1 or 0 options), the result of the tree was that none of the 3 ...
cal1g14's user avatar
  • 101
10 votes
2 answers
9k views

How to evaluate the goodness of fit for survial functions

I am a newcomer to survival analysis, although I have some knowledge in classification and regression. For regression, we have MSE and R square statistics. But how we can say that survival model A ...
floodking's user avatar
  • 323
8 votes
1 answer
4k views

How to split nodes in regression trees

I am looking for a comparison of different regression tree node splitting approaches within the random forest framework. I am looking at the trade-off between ensemble accuracy/reliability (holding ...
Jase's user avatar
  • 2,266
7 votes
2 answers
572 views

Intutition of why Bootstrap aggregating reduces overfitting?

Can somebody give me a non-mathematical intuition why Bootstrap aggregating reduces overfitting? From my point of view, we are not providing any additional information, we are not really enlengthen ...
J3lackkyy's user avatar
  • 655
7 votes
1 answer
11k views

Regression trees - how are splits decided

1) How do we decide if we split into 2 subnodes, or more? Or is it always 2? 2) How do we decide what threshold is the cutoff? Specifically, you have a continuous variable, do you do a binary search, ...
The Baron's user avatar
  • 641
6 votes
1 answer
4k views

Modelling clustered data using boosted regression trees

I'm modelling habitat selection using boosted regression trees (BRTs), which I prefer over linear models for a variety of reasons (modeling complex nonlinear relationships and interactions, ...
Michel's user avatar
  • 111