I am looking to compare various types of models for a dataset, in order to determine which one is the most suitable. They will all be a form of decision tree, from the basic tree to random forests, including oblique trees, random trees, etc. My variable of interest is binary, with a lot more 0s than 1s.

I intend to use many methods for comparison in order to be thorough. (Weighted) cross-validation will of course be included, but what else would you suggest?

If it changes anything, I'm using R, but I am perfectly willing to implement methods myself in the unlikely event that they are not already included in some package.

  • $\begingroup$ Did I get this right: you want a handful of methodological options to add them to your decision tree and then choose one of them later-on? $\endgroup$
    – Klaster
    Jul 28 '15 at 9:45
  • $\begingroup$ No; I must have explained wrong. I have several decision trees, forests, etc..., and I want to select the best one. I'd like to use several tests to be more thorough, so I was asking for suggestions besides your basic cross-validation. $\endgroup$
    – Canisse
    Jul 28 '15 at 14:25