1
$\begingroup$

Let us say that I have a simple data set with 3 columns [A, B, C] and an Output column

where:

A = random number between (0, 1)

B = one of the following vector c('cat', 'dog')

C = either 0 or 1

Output (Y) = either 0 or 1

The conditions for the output column are given below:

if A > 0.5 and C = 1 then Y = 1

else if B = 'cat' then Y = 1

else Y = 0

But, somehow the training data arrived in a way that

wherever (A > 0.5 and C = 1) there is B = 'cat'

and wherever (A <= 0.5 and C = 0) there is B = 'dog'

So, my question is:

If I build a decision tree on this training set - It should be understanding only one pattern (Say, wherever B = 'cat' then Y = 1) and stop growing the tree further as all the data points in the training set accept that rule.

But, If I build a random forest on this training set - As it samples different data points and different features each time (mtry = sqrt(nFeatures)) it should be able to catch the two patterns right?

But, when I wrote the code - in the result the decision tree and random forest are having same performance. Theoretically ensemble models should be better than the base classifiers right? What could be the possible reason? (or) Is there something wrong in my understanding of these models or something wrong in my approach?

Update: Added D Column so that mtry becomes 2 and Modified the code

Here is the R Code:

set.seed(10)
genData = function(nrows) {
    A = runif(nrows)
    B = sample(c('cat', 'dog'), nrows, replace = T)
    C = round(runif(nrows))
    D = sample(c('boy', 'girl'), nrows, replace = T)
    # # Pattern Rules
    # if A > 0.5 and C = 1 then Y = 1 else 0
    # if B = cat then Y2 = 1 else 0
    Y = Y2 = numeric(nrows)
    Y[A > 0.5 & C == 1] = 1
    Y2[B == 'cat'] = 1
    return(data.frame(A, B, C, D, Y, Y2))

}

trainRows = 1000
testRows = 100

train = genData(trainRows)
# Assume that somehow training data has B as Cat and (A > 0.5 and C = 1) only
train = train[(train$Y == 1 & train$Y2 == 1) | (train$Y == 0 & train$Y2 == 0), c("A", "B", "C", "D", "Y")]

# generate test data
test = genData(testRows)
test$Y = ifelse(test$Y == 1 | test$Y2 == 1, 1, 0)

# fit a decision tree
# obviously it should understand that only one pattern exists (either B = 'cat' or [A > 0.5 & C = 1])
library(rpart)
dtFit = rpart(factor(Y) ~ ., data = train)

# fit a random forest
# as it sees different samples of features and data points it should be able to catch both the patterns
library(randomForest)
rfFit = randomForest(factor(Y) ~ ., data = train)
print(table(predict(dtFit, test)[, 2], test$Y))
#    0  1
# 0 41  9
# 1  0  50

print(table(predict(rfFit, test), test$Y))
#    0  1
# 0 41  9
# 1  0  50

Also, Why randomForest is failing to achieve 100% Accuracy (Although it is resulting in same performance as Decision Trees). This is a simple rule based that a powerful ensemble like randomForest should be able to solve right?

$\endgroup$
1
$\begingroup$

In addition to @mariodeng's answer which explains why the random forest trained with default parameters is worse here, here's an explanation why it may not be better than single trees in your experiment anyways:

Aggregated/ensemble models are not universally better than their "single" counterparts, they are better if and only if the single models suffer of instability.

With 1000 training rows and only 3 columns, you are in a comfortable training sample size situation in which even a decision tree may get reasonably stable. (For 3d data you can easily check the variation you have in the assignment of input space to the classes when rerunning the experiment.)

If the predictions of the trees are stable, all submodels in the ensemble return the same prediction and then the prediction of the random forest is just the same as the prediction of each single tree.
So then not only will the overall performance be the same, it will be the same cases that are predicted correctly and wrongly, respectively.

This is the case in your example:

table (predict(dtFit, test) [, 2], predict (rfFit, test))

#     0  1
#  0 46  0
#  1  0 54

why not 100% accurate?

You train on data that is not representative for the test cases: the test cases cover regions of the input space that never appear in the training data. There is no way for a model to know which class (if any - or maybe a 3rd? ...) cases far outside training space should belong to.

Particularly for highly nonlinear partitioning models (such as the decision trees), leaving training space will typically rather sooner than later lead to disaster.

If you plan to train on one class only, you need to look into so-called one-class classifiers which try to establish independent boundaries for each class. One-class classification of your toy data should give you the result that the out-of-training-space cases do not belong to any of the known classes.

Decision trees are a partitioning method, they cannot do one-class classification.

$\endgroup$
  • 1
    $\begingroup$ Aggregated/ensemble models are not universally better than their "single" counterparts, they are better if and only if the single models suffer of instability. - Thanks for this information. $\endgroup$ – Kartheek Palepu Oct 19 '16 at 9:19
  • $\begingroup$ Yes, I have got the same thing. I was wondering why not 100% Accurate. Is it because of the training set is not clearly showing the two patterns? $\endgroup$ – Kartheek Palepu Oct 19 '16 at 9:24
  • $\begingroup$ Excellently put together. Let me grasp more insights on this - If I have to learn the second pattern - I have to retrain the model using the feedback on the test data along with train set. Can this be the conclusion? But, I'm still wondering random forest should build all its trees on first or second patterns itself right? (as the mtry = 2). Is there any tweak that can be done to improve the performance without retraining? or Any other model that can solve this problem? may be xgboost or adaboost? $\endgroup$ – Kartheek Palepu Oct 19 '16 at 9:44
1
$\begingroup$

Just as you mentioned mtry=sqrt(ncol(data)) (with respect to your y column). As you have 3 columns each tree will be grown with 1 column only, as sqrt(3)=1.732051 and 1.732051 will be (integer) cast to 1, so sqrt(3)=1. Therefore, none of the trees will be able to capture any relation between your predictor variables. Not even a single relation for even two variables. A simple partition tree, such as rpart, identifies those relations - as it is using all variables for each split. You can overcome this effect, if you start playing around with the mtry parameter

> rfFit = randomForest(factor(Y) ~ ., data = train, mtry=3)
> print(table(predict(rfFit, test), test$Y))

     0  1
  0 46 13
  1  0 41

Set mtry=3 and each tree will grown similar to rpart, yielding identical results. This way you lose the bagging effect (for the predictor variables, not for the samples), since each tree is grown with the same set of predictors, which can be beneficial in some cases.

$\endgroup$
  • 1
    $\begingroup$ Note that you still have the ensemble effect coming from resampling the rows (unless of course you set ntree = 1). Ensemble models built on all variates can be better than their single submodels - but only if the submodels suffer from instability. $\endgroup$ – cbeleites Oct 19 '16 at 9:07
  • $\begingroup$ Great catch. Why can't mtry = 2 get the patterns? randomForest grows 500 trees with 2 variables at each try. It should be able to catch the pattern right? But, why mtry = 3 works better in this case? $\endgroup$ – Kartheek Palepu Oct 19 '16 at 9:07
  • $\begingroup$ But it does for me!? $\endgroup$ – cbeleites Oct 19 '16 at 9:18
  • 1
    $\begingroup$ No, also randomForests are not magic. => I'll update my answer. $\endgroup$ – cbeleites Oct 19 '16 at 9:27
  • 1
    $\begingroup$ However, the parameter mtry chooses at random the number of columns at every internal node for a tree. So, even is mtry is 1, a tree uses all columns from the data set. $\endgroup$ – Simone Oct 19 '16 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.