Skip to main content

Questions tagged [cart]

'Classification And Regression Trees', also sometimes called 'decision trees'. CART is a popular machine learning technique, and it forms the basis for techniques like random forests and common implementations of gradient boosting machines.

390 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
9 votes
0 answers
657 views

How come the BART results are this good at the 2016 Atlantic causal inference competition?

The famous paper Dorie,2017 shows that BART performs dramatically well in causal inference. In my replication, MSE in BART can be 40% lower than MSE in other machine learning methods. But all machine ...
Ruiyuan Huang's user avatar
8 votes
0 answers
880 views

"Hierarchical" Random forests?

Background I am using Random Forest to classify ~900 objects based on a large number (> 80) predictors. I split these 70:30 for training and testing. The overall model does fairly well, giving an ...
EcologyTom's user avatar
8 votes
0 answers
3k views

Information gain and information gain ratio: Do I have to pick just one?

When splitting attributes while constructing a decision tree, i can use information gain or information gain ratio to try and determine the best value to split the tree on. I'd use information gain ...
casperOne's user avatar
  • 585
5 votes
0 answers
90 views

Is feature importance in sklearn the same as proposed by Breiman 1984?

Breiman 1984 feature importance of a variable $j$ for a regression tree $T$ is: Sklearn regression tree feature importance can be get as follows: Where $\hat{l}_t^2$ is MSE improvement after ...
Davi Américo's user avatar
5 votes
0 answers
593 views

Why use separate trees for each class in multi-class gradient boosting?

Gradient boosted decision trees can be used to solve multi-class classification problems. Friedman (2001) fit $K$ trees on each iteration—one for each class. Multiple GBM implementations also follow ...
user20160's user avatar
  • 32.7k
5 votes
1 answer
2k views

What's the relation between Canonical Correlation Analysis (CCA) and Regression?

I'm wondering if CCA is just a feature transformation method. Can I use it for predicting continuous variables like in regression methods? What I'm doing is to use CCA to transform my training and ...
Shudong's user avatar
  • 571
5 votes
0 answers
181 views

How to mitigate the hierarchical error propagation in tree-structured classification

Suppose we have a multi-class classification problem, where the number of classes $K \geq 3$ We use a tree structure of multiple SVMs to divide and conquer the problem, with one example in the figure ...
nino's user avatar
  • 51
5 votes
1 answer
1k views

C4.5 How to select the split point (threshold) for a Continuous Attribute

Using the "play golf" or "play ball" data (listed at the bottom), to pick the root node we look at Outlook, Temperature, Humidity, and Wind, to see which has the highest GainRatio. Now, Outlook will ...
GreekFire's user avatar
5 votes
0 answers
63 views

Are there any available implementations of density or conditional density tree learning?

I am working on joint and conditional density trees for approximating clique potentials in Bayesian Belief Networks. A brief introduction to topic is available from this paper in case you'd like to ...
mahonya's user avatar
  • 1,111
4 votes
0 answers
88 views

Why are approaches that approximate a random forest with a single decision not more popular?

I understand that random forests yield better performance than standard decision trees, but are less interpretable, because they do not generate a single tree. In this question, several users provided ...
nka5we's user avatar
  • 49
4 votes
0 answers
16 views

Is there a decision tree impurity metric based on maximum of probabilities?

I'm trying to understand impurity metrics in decision tree learning, in particular the Gini impurity. Questioning one of the assumptions of Gini impurity has led me to another impurity measure which ...
A Kubiesa's user avatar
  • 141
4 votes
0 answers
612 views

How do non binary decision trees deal with categorical values that weren't in training?

I've been implementing Random Forest from scratch as a learning exercise. While most algorithms for decision trees seem to deal exclusively in binary yes/no questions, leading to binary trees, ...
Acrid's user avatar
  • 41
4 votes
1 answer
2k views

Random forest splitting rule and importance in package ranger

The package ranger implements random forests in R. Among other things, the function used to fit a random forest allows to choose among several splitting rules, and ...
Augustin's user avatar
  • 233
4 votes
0 answers
1k views

Poisson Deviance - How to Estimate the Optimal Nodes for a Decision Tree

The following is the formulation for how the GBM package in R calculates the loss function and terminal node estimates for gradient boosting with decision trees. My question is generally how are the ...
B_Miner's user avatar
  • 8,780
4 votes
0 answers
2k views

Decision tree with adaboost

Helllo! I'm currently learning the AdaBoost algorithm to use it with Decision Tree. I want to implement everything myself (that's the way I learn - implement everything from scratch and later use redy-...
Animattronic's user avatar
4 votes
0 answers
1k views

Is uplift modeling a solution to the multiple comparisons problem?

If you want to identify a particular user segment for whom an experiment produced some lift or incremental effect over the control treatment, wouldn't it be more direct to do uplift modeling? Uplift ...
Ken Archer's user avatar
4 votes
0 answers
114 views

fitting the tail of a distribution in a regression tree

I have 3 integer valued time series $a_t$, $b_t$ and $y_t$ with $k$ observations. I want to fit $y_t$ with the 2 first, and for that purpose I use a regression tree like this: test all combinations ...
David Bellot's user avatar
4 votes
1 answer
1k views

Optimise Random Forest Model using GridSearchCV in Python

I am working on a classification problem where I am applying various machine learning models. I have used DecisionTreeClassifier from Sklearn on my dataset using the following steps: Calculated alpha ...
tb08's user avatar
  • 123
4 votes
1 answer
4k views

Do classification trees need to consider the correlation between attributes?

In decision tree classification, we use the attribute that splits records, like entropy, as split nodes. Does it need to consider the correlation between attributes?
WeiYuan's user avatar
  • 516
3 votes
0 answers
45 views

Exactly when can I model a two-dimensional region with a 100% accuracy decision tree?

This is a theoritical question about decision trees. I believe that the question is very explicit in the title of the question. My thoughts on this question follow below. If the region we are dealing ...
xyz's user avatar
  • 185
3 votes
0 answers
57 views

What is the "problem of adaptive estimation" that necessitate the development of honest tree?

My question is about the Athey & Imbens (2016) paper. Even though this paper develops honest tree to estimate heterogeneous treatment effect (HTE), in this question I'm only asking about the ...
Heisenberg's user avatar
  • 4,600
3 votes
1 answer
908 views

How are missing values exactly handled in C4.5 decision trees?

I quote Tom M. Mitchell's words on this topic: " A second, more complex procedure is to assign a probability to each of the possible values of A rather than simply assigning the most common value ...
Abhishek Verma's user avatar
3 votes
0 answers
39 views

Literature on Binary regression trees

I'm interested in learning how to use binary regression trees. Therefore I would like to know if there are any seminal papers in the development/application of binary regression trees? As well as if ...
3 votes
1 answer
298 views

Complexity associated with decision trees

According to the sklearn documentation on decision trees: The cost of using the tree (i.e., predicting data) is logarithmic in the number of data points used to train the tree. Could somebody ...
azure31's user avatar
  • 31
3 votes
0 answers
1k views

Random forest [R]: why is my OOB RMSE so much smaller than test RMSE?

I'm doing the kaggle challenge on timetravel predictions where the task is to predict the duration (Y) of a uber trip given some information about the start and end coordinates and the time the trip ...
Amazonian's user avatar
  • 1,544
3 votes
0 answers
2k views

Mathematical definition of the variable importance measure 'increase in node purity' from R randomForests package?

I'm trying to wrap my head around the concept of variable importance (for regression) from the randomForest package in R. I'm trying to find a mathematical ...
Electrino's user avatar
  • 205
3 votes
0 answers
96 views

Quantifying uncertainty of predictions for new data in the regression tree

I used Regression Learner to train my data. I held out 25% of the input for validation and ran different models for training. Based on the results using RMSE and R-squared, I decided to go for the ...
Bobby's user avatar
  • 31
3 votes
0 answers
87 views

Model selection using p-values - tree inference

Suppose I have some i.i.d. normal observations from $\mathbb{R}^f$ with parameters $(\mu, \Sigma)$ and $\Sigma$ is known to be the identity matrix. I have the following hypotheses: $H_0^i$: $\mu_i = ...
user357269's user avatar
3 votes
0 answers
58 views

True or False? Not testing data is needed for CART if there is not future prediction

I am analysing a data set with a relatively a small sample size. For the nature of my data, it is considered already quite large, but for its statistical power it is just in the lower limit. I sampled ...
Cris's user avatar
  • 106
3 votes
0 answers
658 views

How to not overlook rare but important features when preventing over-fitting in a decision tree?

I have a data set where some binary features divide the sample space roughly in half, whereas other features are much less frequent and occur only for 0.0001 - 0.01 of the sample space. However, those ...
user avatar
3 votes
0 answers
195 views

Comparing two split criteria for decision trees

Consider a split criterion for decision trees, which favors splits resulting in groups with as evenly distributed classes as possible. What will be the effect on the resulting decision trees compared ...
Samuel's user avatar
  • 490
3 votes
0 answers
291 views

Classification and regression tree (CART) on large data set

I am trying to approximate a multivariate function $y = f(x_1, ...x_n)$, which I have reason to believe will be well approximated by a classification and regression tree. Some of the variables are ...
nikosd's user avatar
  • 419
3 votes
0 answers
406 views

fragmentation problem in decision tree

I am taking a NLP class, in which it says decision tree has the fragmentation problem. It says ...
WeiChing 林煒清's user avatar
3 votes
0 answers
690 views

Using min leaf size above 1 in random forest/treebagging

Are there any advantages in using a min-leaf-size above 1 in classification with bagged trees(/random forests)? I would imagine that it could make the classification more robust as some area in the ...
fixingstuff's user avatar
3 votes
0 answers
3k views

How to split a decision tree when information gains of all attributes are zero?

The textbook tells us that we should choose an attribute with the maximum information gain to split a decision tree. My question is what if all information gains are zero? Should we stop splitting or ...
azure's user avatar
  • 31
3 votes
0 answers
295 views

Tricks for a very fast implementation of Random Forest

I am implementing my own Random (regression) Forest algorithm and am looking for tricks to speed up the estimation of forests on large datasets. So far I have implemented three main tricks: 1) Use a ...
Jase's user avatar
  • 2,266
3 votes
0 answers
185 views

Random classification forests for extremely sparse response variables

I have a response variable that can be $A,B,C$. It is very sparse, meaning 99% of the sample is $B$ and the rest is approximately evenly divided between $A$ and $C$. How do I predict this variable in ...
Jase's user avatar
  • 2,266
3 votes
0 answers
2k views

Weka J48 decision tree problem

I have a CSV dataset which contains mean (Numeric), spread (Numeric), review (string), ...
kishore's user avatar
  • 39
3 votes
0 answers
1k views

decision tree using user defined split function in rpart: No splits returned when tree is run

I've written a user defined splitting function to use with rpart, its returning a 'vector of goodness', but the tree that is returned never has any splits, just one node. using the anova method on ...
justin cress's user avatar
3 votes
1 answer
3k views

Decision Tree for Time Series Anomaly Detection

I have learned recently to train decision trees in R with data. Now I have a problem in which the data is a time series and I would like to use the same approach to detect when the time series ...
Ambesh's user avatar
  • 333
2 votes
0 answers
378 views

How is Gini impurity related to accuracy when predicting the majority class?

For simplicity, consider the binary case, where we have a set of elements with each element belonging to one of two classes (0 or 1). Let p(j) be the proportion of ...
alex76's user avatar
  • 21
2 votes
1 answer
163 views

Gini impurity greedily optimises a loss function in decision trees

I am trying to understand how the Gini criterion for decision decision tree construction actually greedily optimises a loss function. The Gini impurity, sometimes also called Gini index, for a region (...
ngmir's user avatar
  • 339
2 votes
0 answers
47 views

Confusion about the code for choosing "stumps" in Adaboost algorithm

(I actually asked the following question on Stack Overflow recently: https://stackoverflow.com/questions/76842431/confusion-about-the-code-for-choosing-stumps-in-adaboost-algorithm but then I ...
Richard's user avatar
  • 121
2 votes
0 answers
41 views

Can you use regression trees for classification tasks in random forest?

I've been playing around with random forest algorithm to classify a binary Y vector using classification and regression trees. Classification trees output class probability and regression trees an ...
Mirko Pavicic's user avatar
2 votes
0 answers
57 views

How to identify differences between two decision trees

I have two decision trees that gives each individual a rating based on the same 4 parameters (two quantitative and one qualitative). They are supposed to be exactly the same, but give different ...
Dimitri's user avatar
  • 41
2 votes
0 answers
281 views

Can XGBoost handle a custom objective where the 2nd derivative can be negative?

I am going through the introduction to XGBoost page, and there is a section where they derive the optimal value of the leaf node, for a given tree structure. To quote the specific section, In this ...
Bridgeburners's user avatar
2 votes
0 answers
156 views

Directed Acyclic Graph including a categorical variable with 20 levels

Is it possible within causal inference using DAGs to sensibly include a categorical variable with 20 levels? I have seen that regression trees can be used in this situation but not in combination with ...
ReadBeard's user avatar
  • 333
2 votes
0 answers
659 views

Are binary splits always better than multi-way splits in decision trees?

I'm trying to devise a decision tree for classification with multi-way split at an attribute but even though calculating the entropy for a multi-way split gives better information gain than a binary ...
dorito's user avatar
  • 21
2 votes
0 answers
319 views

Interpretation of cv.tree plot

I am conducting research using regression trees and I was hoping someone here could help me understand the following cv.tree plot. I understand that the x-axis (bottom) represents the size of the ...
Ben Chip's user avatar
2 votes
0 answers
88 views

Classification algorithm find closest observations

I am currently using Light GBM (Gradient Boosting) in Python, but this question can apply to other classification algorithms. In order to improve my model's explainability I would like to be able to ...
Mustard Tiger's user avatar

1
2 3 4 5
8