I am attempting to make some regression trees with many potential independent variables which comprise both categorical and continuous data types of widely varying scales
i have been using a few different methods of modelling with different validation methods
ANNs, Decision trees, Boosted trees.
Validation with hold back, 3 way hold back and k-fold
i have decided to focus on decision trees due to the simple ability to convey results to others. I have also decided that due to holdback validation producing some quite different models dependent on which members are used for the validation that the dataset is too small for this
What i am left with is k-fold decision tree which appears to split too far despite no k-fold validation drop off . I was shown a method (by a SAS rep) where a variable is inserted into the model making process that is essentially junk to help identify this
using this method i have found that the 'junk' variable gets incorporated into the model sometimes quite early on and splits are performed beyond this. Is it valid to prune the tree back to remove these to be left with a more true model?
incidentally, the hold back validation method produces similar models (in terms of number of splits) as this k-fold pruned model
sorry for the long read, help would be appreciated!
Edt: I would like to add that when using 'junk' data in holdback it never gets incorporated into the model due to the early stopping that occurs in splitting variables