0
$\begingroup$

I am attempting to make some regression trees with many potential independent variables which comprise both categorical and continuous data types of widely varying scales

i have been using a few different methods of modelling with different validation methods

ANNs, Decision trees, Boosted trees.

Validation with hold back, 3 way hold back and k-fold

i have decided to focus on decision trees due to the simple ability to convey results to others. I have also decided that due to holdback validation producing some quite different models dependent on which members are used for the validation that the dataset is too small for this

What i am left with is k-fold decision tree which appears to split too far despite no k-fold validation drop off . I was shown a method (by a SAS rep) where a variable is inserted into the model making process that is essentially junk to help identify this

using this method i have found that the 'junk' variable gets incorporated into the model sometimes quite early on and splits are performed beyond this. Is it valid to prune the tree back to remove these to be left with a more true model?

incidentally, the hold back validation method produces similar models (in terms of number of splits) as this k-fold pruned model

sorry for the long read, help would be appreciated!

Edt: I would like to add that when using 'junk' data in holdback it never gets incorporated into the model due to the early stopping that occurs in splitting variables

$\endgroup$
1
  • $\begingroup$ Added; Y is normally distributed, consists of approx 1000 members and 100 potential Xs (after severe correlation has been removed) $\endgroup$
    – Samuel
    Jan 2, 2014 at 14:08

1 Answer 1

1
$\begingroup$

Single trees have severe difficulties with predictive accuracy and stability. Interpretability is largely an illusion. Single trees are only interpretable because they are wrong. Do are dealing with multiplicity problems of monumental magnitude, effectively estimating hundreds of parameters. By the way please state the sample size and distribution of $Y$.

$\endgroup$
1
  • $\begingroup$ I have added the requested info, when using boosted methods generally what i get is the lower portions of the single trees remains very similar. $\endgroup$
    – Samuel
    Jan 2, 2014 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.