5
votes
0answers
98 views

Random forest on multi-level/hierarchical-structured data

I am quite new to machine learning, CART-techniques and the like, and I hope my naivete isn't too obvious. How does Random Forest handle multi-level/hierarchical data structures (for example when ...
0
votes
0answers
36 views

ML algorithm to find optimal control parameter

I have a training dataset $(X, y) \rightarrow z$. Where $X$ is an $n$th dimensional real vector, $y$ is an integer number in $\{1, 2, 3\}$, and $z$ is a real number. I am looking for machine learning ...
2
votes
1answer
34 views

In decision tree construction, can a good splitter have low information gain?

I have a data set with a candidate splitter variable that is a natural choice from the business perspective. It has two values, and the distributions of the target when conditioned on the two values ...
0
votes
0answers
14 views

Choosing the values of a proper subset of features to maximise regression tree output

Suppose I have a regression tree and feature set $X$. Suppose that the feature set is composed of $X:=\{X_0,X_1,...,X_{100}\}$, where each $X_i \sim N(0,\sigma^2)$. Suppose that ...
3
votes
1answer
42 views

Why the trees generated via bagging are identically distributed?

I have problem in intuitive understanding of following arguement: "The trees generated via bagging are identically distributed, thus the expectation of the average of a set of trees is the same as ...
0
votes
0answers
16 views

Early split decision criteria for fast random (regression) forest estimation

Suppose I am on a node in a $regression$ tree and I am using running estimates of $\sum_{i \in Region_1} (y_i - mean(y_i)_{Region1})^2$ (and the same for Region 2) to determine whether to split the ...
3
votes
0answers
37 views

Standard deviation in regression trees

In a regression tree, it is often assumed that each leaf is a Gaussian distribution $\mathcal{N}(\mu_i, \sigma)$, where $i$ is the index the leaf. Is $\sigma$ calculated as the standard deviation ...
1
vote
1answer
65 views

How to incorporate constraints in random forest output

Suppose I am doing random forest classification of labels $A$,$B$,$C$,$D$. There is some theoretical ordering to this output such that when $A$ is more likely than $B$, $B$ is also more likely than ...
0
votes
1answer
48 views

Can we remove trees from a random forest with poor OOB error to improve generalisation?

My objective is to improve out of sample generalization of my random forest while holding the number of trees constant. Suppose that I am only allowed to use $n$ trees on the out of sample data but ...
0
votes
1answer
83 views

How to interpret scikit learn classification tree?

I'm currently trying to work with scikit-learn classification tree. I followed the example on iris dataset : http://scikit-learn.org/stable/modules/tree.html and everything is working fine. I do ...
1
vote
1answer
87 views

Theoretical error bounds of classification and regression trees

So, some algorithms were motivated by theoretical work, such as in the case of boosting. Adaboost was introduced as an algorithm for solving the hypothesis boosting problem. The bounds on the training ...
4
votes
2answers
85 views

The first principal component becomes irrelevant

I did run PCA on 17 quantitative variables in order to obtain a smaller set of variables that is principal components to be used in supervised machine learning for classifying instances into two ...
1
vote
1answer
148 views

Decision trees variable (feature) scaling and variable (feature) normalization (tuning) required in which implementations?

In many machine learning algorithms, feature scaling (aka variable scaling, normalization) is a common prepocessing step Wikipedia - Feature Scaling -- this question was close Question#41704 - How and ...
0
votes
0answers
17 views

Is there an approximate relationship between n.trees and training set's size for gradient boosting?

Can anyone tell me if there is an approximate relationship between n.trees and training set's size for gradient boosting?
0
votes
2answers
164 views

Decision tree for output prediction

I have satellite data that provides radiance which I use to compute the Flux (using surface and cloud info). Now using a regression method, I can develop a mathematical model directly relating ...
3
votes
0answers
80 views

Large n small p regression - Machine Learning

In the area of machine learning, most of the algorithms are intended for small n large p problems. I am familiar with the statistical techniques of PCA, etc but was wondering what algorithms are ...
0
votes
0answers
91 views

Decision trees vs Decision list

Is a decision list same as a decision tree? Outputs from SPSS modeler for decision lists and the theories that I am seeing on Google link them together but I am not sure if I am understanding it ...
1
vote
0answers
134 views

Deciding attributes for decision trees

I'm a complete beginner when it comes to R and decision trees, but I was asked to take a look at this to see if this was a viable solution for my data. So please excuse me if I say completely wrong. ...
2
votes
0answers
82 views

Is there a theoretical basis for the shrinkage used in Boosted Regression Trees?

In Gradient Boosted Regression Trees, a shrinkage $\nu$ is often applied as: $$ f_t(x) \leftarrow f_{t-1}(x) + \nu h(x)$$ where $h$ is the regression tree learned by fitting the tree to the gradient. ...
2
votes
1answer
177 views

Building the dataset for Random Forest training procedure

I should use the bagging (bootstrap aggregating) technique in order to train a random forest classifier. I read here the description of this learning technique, but I have not figured out how I ...
1
vote
0answers
74 views

Determining optimal height for regression tree

I have a data set of approximately 400,000 records (for those of you who know, the data set is the one provided by yahoo for their yahoo learning to rank challenge). From this data set I learn a ...
2
votes
1answer
1k views

What is “feature space”?

What is the definition of "feature space"? For example, When reading about SVMs, I read about "mapping to feature space". When reading about CART, I read about "partitioning to feature space". I ...
7
votes
0answers
526 views

Would a Random Forest with multiple outputs be possible/practical?

Random Forests (RFs) is a competitive data modeling/mining method. An RF model has one output -- the output/prediction variable. The naive approach to modeling multiple outputs with RFs would be to ...
5
votes
1answer
130 views

Is there a method for constructing decision trees that takes account of structured/hierarchical/multilevel predictors?

Is there a method for constructing decision trees that takes account of structured/hierarchical/multilevel predictors, that would allow me to impose domain knowledge or constraints on interactions for ...
3
votes
1answer
167 views

Should I use decision trees to predict user preferences?

I'm designing a web service that will predict and recommend new items a user might like based on their expressed preferences on previous items (simple thumbs up/down interface). I was told to look ...
6
votes
4answers
963 views

Benchmark dataset for decision tree algorithm

I'm implementing a decision tree algorithm, and I'd like to get a feel for how it performs relative to other implementations. Can anyone recommend popular datasets for training and testing decision ...
1
vote
0answers
57 views

Relationship between the characteristics of training data set and built decision tree

I have a training data set for a binary classification problem. There exist two possible scenarios, one is that all of the training data set are labeled as positive; another one is that the training ...
1
vote
0answers
259 views

Obtaining resampling based estimates of prediction error in boosted regression tree model

I try to use the gbm.fit() function for a boosted regression tree model implemented in the R package gbm. To investigate e.g., the bootstrapped prediction error and ...
4
votes
2answers
149 views

Incorporating seasonality into CART models

The problem I am trying to solve it predicting sales for an item for the next $n$ weeks. Obviously, seasonality is a major factor for such predictions. If we use a time series based model, then we ...
5
votes
3answers
1k views

Are decision trees almost always binary trees?

Nearly every decision tree example I've come across happens to be a binary tree. Is this pretty much universal? Do most of the standard algorithms (C4.5, CART, etc.) only support binary trees? From ...
30
votes
2answers
5k views

Conditional inference trees vs traditional decision trees

Can anyone explain the primary differences between conditional inference trees (ctree from party package in R) compared to the ...
4
votes
1answer
260 views

Do infrequent examples screw up classifiers? If so, when is it okay to remove the infrequent examples from the data?

It's hard to think of a more eloquent way of phrasing this question - I'm basically wondering if a classifier trained on data where examples of some of the classes are infrequent/rare would be a bad ...
8
votes
2answers
620 views

Are there any libraries available for CART-like methods using sparse predictors & responses?

I'm working with some large data sets using the gbm package in R. Both my predictor matrix and my response vector are pretty sparse (i.e. most entries are zero). I was hoping to build decision trees ...
3
votes
1answer
2k views

What is the difference between empirical variance and variance?

As far as I know variance is calculated as $$\text{variance} = \frac{(x-\text{mean})^2}{n}$$ while $$\text{Empirical Variance} = \frac{(x-\text{mean})^2}{n(n-1)} $$ Is it correct? Or is there ...
15
votes
6answers
1k views

Alternatives to classification trees, with better predictive (e.g: CV) performance?

I am looking for an alternative to Classification Trees which might yield better predictive power. The data I am dealing with has factors for both the explanatory and the explained variables. I ...
10
votes
4answers
2k views

What is the weak side of decision trees?

Decision trees seems to be a very understandable machine learning method. Once created it can be easily inspected by a human which is a great advantage in some applications. What are the practical ...
13
votes
2answers
1k views

How does random forest generate the random forest

I am not an expert of random forest but I clearly understand that the key issue with random forest is the (random) tree generation. Can you explain me how the trees are generated? (i.e. What is the ...