5
$\begingroup$

I'm analyzing decision trees on a regression problem with 12 attributes, a class attribute that can have values between 1-10, and 6497 records. Here is the data. I am using 10 fold cross-validation for my analysis. My analysis of the dataset is below:

Pruned   Confidence   Tree size   train%   RMSE-train   test%   RMSE-test   
------|-------------|----------|---------|------------|-------|------------
Yes   | 0.125       | 885      | 88.74   | 0.15       | 58.33 | 0.31
Yes   | 0.25        | 1032     | 91.19   | 0.13       | 58.75 | 0.31
No    | --          | 1135     | 92.02   | 0.13       | 58.82 | 0.32

Based on my analysis, I have a few questions regarding decision trees to further clear my understanding of them

  • Do decision trees usually have a low success rate while predicting values in a class that may have more than a few values (in my case 1 - 10)
  • Is it fair to assume that if the class is discrete with 10 possible values, then some attributes will appear multiple times in the tree as compared to when the class may only have two values?
  • Since preference bias of trees is to choose shorter non complex trees, if the resulting models tree size is huge (885 even after post pruning)...is it expected that the success rate will be low?
  • If I'm already analyzing my data using cross validation, is there an added benefit in running the learner on different % of test and train sets? i.e. run with 10% train and 90% test then 20% train and 80% test..etc.
$\endgroup$
1
3
+50
$\begingroup$

Do decision trees usually have a low success rate while predicting values in a class that may have more than a few values (in my case 1 - 10)

Depends mostly on the specific dataset / specific problem than decision trees themselves.

Is it fair to assume that if the class is discrete with 10 possible values, then some attributes will appear multiple times in the tree as compared to when the class may only have two values?

The re-usability of features comes more down to the features' relation to the classes than just the number of possible states your output class can take. I expect to see a few features be much more useful than the rest of features in any problem of the sort. In decision trees, 'more useful' manifests as more higher quality decision boundaries.

Since preference bias of trees is to choose shorter non complex trees, if the resulting models tree size is huge (885 even after post pruning)...is it expected that the success rate will be low?

I haven't personally seen useful trees that big, but it sounds like you're framing the experiment properly. So if trees that big are proving to be better in your cross validation, then it sounds like it's okay. Consider throwing a few different models at the data and get a more concrete baseline for this dataset.

If I'm already analyzing my data using cross validation, is there an added benefit in running the learner on different % of test and train sets? i.e. run with 10% train and 90% test then 20% train and 80% test..etc.

Seeing how low of a train percentage you can use and still get about-the-same results would be an interesting way to see how hard the data is to learn. I.e., finding the low threshold where you start seeing a few folds perform significantly worse than the rest will give you a way to introspect how consistent the learned patterns are in the dataset you're working with.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.