0
votes
0answers
7 views

VC-Dimension of n-node binary decision tree in N-dimension feature space

Given input feature space $\mathcal{X} =\{0, 1\}^N$ and output label space $\mathcal{Y}=\{0,1\}$ , prove that the VC-dimension of a binary decision tree with $n$ nodes is in $O(n\text{log}N)$. I've ...
2
votes
0answers
47 views

How to split a decision tree when information gains of all attributes are zero?

The textbook tells us that we should choose an attribute with the maximum information gain to split a decision tree. My question is what if all information gains are zero? Should we stop splitting or ...
2
votes
0answers
16 views

mob model tree algorithm

I am trying to figure out the inner workings of the mob function in the party package. I can't figure out how the splitting variable is selected when it is a categorical variable. In the publications ...
0
votes
0answers
5 views

number of nodes in an unpruned decision tree

What is the number of nodes in an unpruned decision tree that is trained using n samples and that grows until there is only one sample in each leaf? I would like to know if there is a formula to ...
0
votes
0answers
25 views

Sample Weights for classification using Gradient-Boosted trees?

How can "weights" be given to different samples according to their relative importance while using Gradient boosted decision trees for classification? How does the ...
1
vote
1answer
46 views

Post hoc selection of important features in random forest?

I want to guarantee a parsimonious random forest (few features used). What are methods to do this? It was suggested to me to get the feature importance after the model was created, and then create a ...
0
votes
0answers
5 views

Design a feature with time and presence information

Context: I am working on a decision tree classifier, trying to classify businesses as to whether they are likely to have an event occur (default) in the next 90 days. One input I get is whether, and ...
0
votes
0answers
22 views

Improving sentence segmentation in NLTK

I have been looking into problem of sentence segmentation lately. I have been referring to NLTK's book for this purpose. I followed their procedure to segment sentences presented here: ...
0
votes
0answers
69 views

Decision tree in R

I am new to machine learning in R. This is my data set. ...
1
vote
0answers
53 views

How do i estimate the Weights of the predictions assigned to each of the tree in GBM using R? How does GBM split nodes?

I ran a GBM model in R with loss function as bernoulli and n.trees=1000. I want to see the weights assigned to the predictions coming from 1000 trees. Is there any command in R that does that? How ...
0
votes
1answer
68 views

encrypted data on CART, ID3

Some data are confidential such as patient data. Therefore sometimes companies does not want to give original patient data instead they first encrypt it(for instance with SHA1) and then give. If we ...
2
votes
1answer
192 views

Regression Trees / Boosted Regression Trees for Tweedie Distribution in R

I am currently working at work on a project that attempts to predict an environmental change variable. I am personally not a huge fan of the project, but I still want to do the best job possible. ...
3
votes
1answer
95 views

Benefits of CART over ID3 algorithm

When building decision trees over a dataset that generates nodes with bad purity, is there any benefit of using the CART algorithm over the iterative dichotomizer 3 (ID3) algorithm?
0
votes
0answers
19 views

If my feature of interest has many values should I pre-process them into groups?

I'm quite new to machine learning, pattern recognition, statistics, etc. but I'm trying to wrap around how a machine learning system would interpret data that is something like this: row0-> ...
0
votes
1answer
79 views

Get probability distribution from decision tree

I'm implementing decision tree based on CART algorithm and I have a question. Now I can classify data, but my task is not only classify data. I want have a probability of right classification in end ...
1
vote
1answer
53 views

Decision trees for advertising data

Assuming a dataset with the following attributes: Date (truncated), f1 ... fn, ...
6
votes
1answer
226 views

Random forest on multi-level/hierarchical-structured data

I am quite new to machine learning, CART-techniques and the like, and I hope my naivete isn't too obvious. How does Random Forest handle multi-level/hierarchical data structures (for example when ...
0
votes
0answers
48 views

ML algorithm to find optimal control parameter

I have a training dataset $(X, y) \rightarrow z$. Where $X$ is an $n$th dimensional real vector, $y$ is an integer number in $\{1, 2, 3\}$, and $z$ is a real number. I am looking for machine learning ...
2
votes
1answer
69 views

In decision tree construction, can a good splitter have low information gain?

I have a data set with a candidate splitter variable that is a natural choice from the business perspective. It has two values, and the distributions of the target when conditioned on the two values ...
0
votes
0answers
16 views

Choosing the values of a proper subset of features to maximise regression tree output

Suppose I have a regression tree and feature set $X$. Suppose that the feature set is composed of $X:=\{X_0,X_1,...,X_{100}\}$, where each $X_i \sim N(0,\sigma^2)$. Suppose that ...
3
votes
1answer
49 views

Why the trees generated via bagging are identically distributed?

I have problem in intuitive understanding of following arguement: "The trees generated via bagging are identically distributed, thus the expectation of the average of a set of trees is the same as ...
0
votes
0answers
29 views

Early split decision criteria for fast random (regression) forest estimation

Suppose I am on a node in a $regression$ tree and I am using running estimates of $\sum_{i \in Region_1} (y_i - mean(y_i)_{Region1})^2$ (and the same for Region 2) to determine whether to split the ...
3
votes
0answers
60 views

Standard deviation in regression trees

In a regression tree, it is often assumed that each leaf is a Gaussian distribution $\mathcal{N}(\mu_i, \sigma)$, where $i$ is the index the leaf. Is $\sigma$ calculated as the standard deviation ...
2
votes
1answer
97 views

How to incorporate constraints in random forest output

Suppose I am doing random forest classification of labels $A$,$B$,$C$,$D$. There is some theoretical ordering to this output such that when $A$ is more likely than $B$, $B$ is also more likely than ...
0
votes
1answer
85 views

Can we remove trees from a random forest with poor OOB error to improve generalisation?

My objective is to improve out of sample generalization of my random forest while holding the number of trees constant. Suppose that I am only allowed to use $n$ trees on the out of sample data but ...
0
votes
1answer
220 views

How to interpret scikit learn classification tree?

I'm currently trying to work with scikit-learn classification tree. I followed the example on iris dataset : http://scikit-learn.org/stable/modules/tree.html and everything is working fine. I do ...
1
vote
1answer
92 views

Theoretical error bounds of classification and regression trees

So, some algorithms were motivated by theoretical work, such as in the case of boosting. Adaboost was introduced as an algorithm for solving the hypothesis boosting problem. The bounds on the training ...
4
votes
2answers
89 views

The first principal component becomes irrelevant

I did run PCA on 17 quantitative variables in order to obtain a smaller set of variables that is principal components to be used in supervised machine learning for classifying instances into two ...
1
vote
1answer
276 views

Decision trees variable (feature) scaling and variable (feature) normalization (tuning) required in which implementations?

In many machine learning algorithms, feature scaling (aka variable scaling, normalization) is a common prepocessing step Wikipedia - Feature Scaling -- this question was close Question#41704 - How and ...
0
votes
2answers
267 views

Decision tree for output prediction

I have satellite data that provides radiance which I use to compute the Flux (using surface and cloud info). Now using a regression method, I can develop a mathematical model directly relating ...
3
votes
0answers
96 views

Large n small p regression - Machine Learning

In the area of machine learning, most of the algorithms are intended for small n large p problems. I am familiar with the statistical techniques of PCA, etc but was wondering what algorithms are ...
1
vote
0answers
185 views

Deciding attributes for decision trees

I'm a complete beginner when it comes to R and decision trees, but I was asked to take a look at this to see if this was a viable solution for my data. So please excuse me if I say completely wrong. ...
3
votes
1answer
149 views

Is there a theoretical basis for the shrinkage used in Boosted Regression Trees?

In Gradient Boosted Regression Trees, a shrinkage $\nu$ is often applied as: $$ f_t(x) \leftarrow f_{t-1}(x) + \nu h(x)$$ where $h$ is the regression tree learned by fitting the tree to the gradient. ...
2
votes
1answer
192 views

Building the dataset for Random Forest training procedure

I should use the bagging (bootstrap aggregating) technique in order to train a random forest classifier. I read here the description of this learning technique, but I have not figured out how I ...
1
vote
0answers
77 views

Determining optimal height for regression tree

I have a data set of approximately 400,000 records (for those of you who know, the data set is the one provided by yahoo for their yahoo learning to rank challenge). From this data set I learn a ...
2
votes
1answer
2k views

What is “feature space”?

What is the definition of "feature space"? For example, When reading about SVMs, I read about "mapping to feature space". When reading about CART, I read about "partitioning to feature space". I ...
8
votes
1answer
725 views

Would a Random Forest with multiple outputs be possible/practical?

Random Forests (RFs) is a competitive data modeling/mining method. An RF model has one output -- the output/prediction variable. The naive approach to modeling multiple outputs with RFs would be to ...
6
votes
1answer
159 views

Is there a method for constructing decision trees that takes account of structured/hierarchical/multilevel predictors?

Is there a method for constructing decision trees that takes account of structured/hierarchical/multilevel predictors, that would allow me to impose domain knowledge or constraints on interactions for ...
3
votes
1answer
190 views

Should I use decision trees to predict user preferences?

I'm designing a web service that will predict and recommend new items a user might like based on their expressed preferences on previous items (simple thumbs up/down interface). I was told to look ...
6
votes
4answers
1k views

Benchmark dataset for decision tree algorithm

I'm implementing a decision tree algorithm, and I'd like to get a feel for how it performs relative to other implementations. Can anyone recommend popular datasets for training and testing decision ...
1
vote
0answers
58 views

Relationship between the characteristics of training data set and built decision tree

I have a training data set for a binary classification problem. There exist two possible scenarios, one is that all of the training data set are labeled as positive; another one is that the training ...
1
vote
0answers
289 views

Obtaining resampling based estimates of prediction error in boosted regression tree model

I try to use the gbm.fit() function for a boosted regression tree model implemented in the R package gbm. To investigate e.g., the bootstrapped prediction error and ...
4
votes
2answers
167 views

Incorporating seasonality into CART models

The problem I am trying to solve it predicting sales for an item for the next $n$ weeks. Obviously, seasonality is a major factor for such predictions. If we use a time series based model, then we ...
6
votes
3answers
2k views

Are decision trees almost always binary trees?

Nearly every decision tree example I've come across happens to be a binary tree. Is this pretty much universal? Do most of the standard algorithms (C4.5, CART, etc.) only support binary trees? From ...
34
votes
2answers
7k views

Conditional inference trees vs traditional decision trees

Can anyone explain the primary differences between conditional inference trees (ctree from party package in R) compared to the ...
4
votes
1answer
272 views

Do infrequent examples screw up classifiers? If so, when is it okay to remove the infrequent examples from the data?

It's hard to think of a more eloquent way of phrasing this question - I'm basically wondering if a classifier trained on data where examples of some of the classes are infrequent/rare would be a bad ...
8
votes
2answers
799 views

Are there any libraries available for CART-like methods using sparse predictors & responses?

I'm working with some large data sets using the gbm package in R. Both my predictor matrix and my response vector are pretty sparse (i.e. most entries are zero). I was hoping to build decision trees ...
4
votes
1answer
3k views

What is the difference between empirical variance and variance?

As far as I know variance is calculated as $$\text{variance} = \frac{(x-\text{mean})^2}{n}$$ while $$\text{Empirical Variance} = \frac{(x-\text{mean})^2}{n(n-1)} $$ Is it correct? Or is there ...
16
votes
6answers
2k views

Alternatives to classification trees, with better predictive (e.g: CV) performance?

I am looking for an alternative to Classification Trees which might yield better predictive power. The data I am dealing with has factors for both the explanatory and the explained variables. I ...
11
votes
4answers
3k views

What is the weak side of decision trees?

Decision trees seems to be a very understandable machine learning method. Once created it can be easily inspected by a human which is a great advantage in some applications. What are the practical ...