1
$\begingroup$

Working with decision tree and I have couple of questions:

  1. Should I always do random forest before or I can just do the decision tree, skip the random forest part?
  2. Should I always have a training data set and test set? Or, I can run the model just on the orginal data?
  3. What method is suitable if you don't have any factors in the data set all are numeric?
  4. Root node error: here we have 101653/118 = 861.46, seemse a lot? Is it now overfitted? Possible solutsions?

Added the code and giving background on my data:

  • ob.prcnt is value % of eaten insects in specific place

  • I also have 11 variables that are all specific percentage of habitat in sampling place.

I would like to see how these habitat amounts affect the insects eaten %.
A good guide on decision tree and how to do them in R is appreciated!

> fit.tree14<- rpart(ob.prcnt ~ ., data=P14_Q1)
> #fit.tree14
> #summary(fit.tree14)
> rpart.plot(fit.tree14, extra=1, type=2, digits=3,
+            clip.right.labs=TRUE, under=TRUE, 
+            branch=1, tweak = 1.1, gap=6, space=1) #, main = "2015", cex.main = 1.5)
> 
> #rpart.rules(fit.tree)
> printcp(fit.tree14)

Regression tree:
rpart(formula = ob.prcnt ~ ., data = P14_Q1)

Variables actually used in tree construction:
[1] Agricultural.land.excluding.permanent.grassland Semi.natural.habitat                           
[3] Woody.linear                                   

Root node error: 101653/118 = 861.46

n= 118 

        CP nsplit rel error xerror    xstd
1 0.096492      0   1.00000 1.0270 0.12096
2 0.015656      2   0.80702 0.9749 0.12522
3 0.011298      4   0.77570 1.0441 0.13470
4 0.010000      5   0.76440 1.0522 0.13746
$\endgroup$
1
  • $\begingroup$ As for your question on a good guide on Decision Trees, there's an innumerable amount of guides on the web so it's hard to recommend a specific one. I feel like software documentation pages can be quite terse and do not provide lots of insight. DTs and RFs do not require all that much previous knowledge so it might be worthwhile to look at some research papers. There are reviews/introductions like this one or specifically for your field (e.g. for bioinformatics) $\endgroup$
    – ngmir
    Apr 27, 2023 at 11:18

1 Answer 1

2
$\begingroup$

Random forest is one way (things like gradient boosted decision trees - e.g. XGBoost/LightGBM are other ways - which tend to have often have slightly better prediction performance) of building trees that in combination ("as an ensemble") can be much more complex than a single tree without overfitting. By "without overfitting" I don't mean that they cannot overfit, but rather than it turns out to be easier to regularize the process of building an ensemble of trees (by tuning various hyperparameters of the process) so that they don't overfit (as assessed by e.g. cross-validation). Whether you want these more complex models is mostly a function of what's your goal. Often, you will need the more complex ones, if you primarily want to optimize performance and interpretation is not the main goal (if interpretation is very important various interpretation tools exist, but also there's research into reasonably performant ways of building much simpler models such as CORELS).

I.e. whether you always want to do random forest vs. just a decision tree really depends on what you want to achieve and the context.

Regarding training and test (and validation set), you usually want (at least) three things:

  • to train your model based on data: for this you need a training set
  • to choose hyperparameters of your model: you usually need to do this on data that's not your training set (otherwise you tend to overfit), one solution is a separate "validation set" (or cross-validation to be more efficient) - once you've chosen your hyperparameters, you can re-train on the combined training + validation set with the chosen hyperparameters
  • to know how well your model is working: if you use the data you trained on or on which you choose the hyperparameters (aka the validation set), then you are potentially misleading yourself (and others, often very, very badly - I've seen flawed evaluations on the training data that suggested near perfect performance when the model was just producing useless garbage), so if you want to know how well your model works you need new data to evaluate it on

For the last two points, the considerations outlined here apply. I.e. you don't just need any data or to thoughtlessly randomly split your data, you really need to think about this.

If you don't have factors in the data, this does not change too much except that it makes your life easier. Purely numeric data are easier to deal with for many algorithms and a lot of effort (e.g. target encoding, creating embeddings etc.) goes into how to represent categorical features for models that don't deal with them so well.

$\endgroup$
5
  • $\begingroup$ Just want to throw in a related concept for completeness' sake: Besides the usual train/test/validation splits applicable to any methods, for ensemble methods and Random Forests in particular, there's the "Out-of-Bag Estimate". Each tree in the forest is trained on only a subset of the training data ("bootstrapping"). Feeding examples through the trees for whose training they have not been used and analysis the overall error thereof is (roughly) the "Out-of-Bag" Performance/Score. $\endgroup$
    – ngmir
    Apr 27, 2023 at 11:12
  • $\begingroup$ Thank you! You say that it is easier with numeric data, but how do deal with overfitting here, do I need to be concerned about overfitting? For example the data I am working on have a really spesific value for each sampling point (regarding the ob.precnt and the landscape % around that area), and if I run desicion tree, does it just create the tree on the values? Do I need to validate the tree some how? Or just say that based on the habitat 1 where the precentage of that in thea area equaled to 30% of the the tree predicted ob.precnt to be 20% etc.? $\endgroup$
    – Sisi
    May 2, 2023 at 7:09
  • $\begingroup$ Yes, you very much have to be worried about overfitting. A way to reduce it is cross-validation (CV): You split the data into e.g. 10 parts and then 10x use 9 parts for training + 1 part for "validation" with each part being the "validation" part once. That lets you reduce overfitting by looking at what the model does on data it was not trained on. You choose settings for the model to optimize performance on the validation parts of CV (even that has some potential for overfitting to the out-of-fold parts, but is a lot of better than looking at performance on the data you trained the model on). $\endgroup$
    – Björn
    May 2, 2023 at 7:29
  • $\begingroup$ Thank you! Can you provide a tutorial/script/guid, for this? It would be much appreciated $\endgroup$
    – Sisi
    May 2, 2023 at 8:12
  • $\begingroup$ E.g. this book (pdf made available by the authors online): statlearning.com in particular chapter 5, but the whole book is nice. From a slightly more practical perspective, this book by a Kaggle grandmaster is nice: github.com/abhishekkrthakur/approachingalmost $\endgroup$
    – Björn
    May 2, 2023 at 10:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.