I'm trying to compare two classifiers using cross-validation. For one of the classifiers I'd like to use a validation set to tune a parameter. However, for the other one, I just want to use the training set and no validation set. Writing my own cross-validation loop, I'm not sure how to implement this. Is it correct to use the train set to estimate one of the classifiers and only 80% of the train set to estimate the other? This way, I could then tune one of the models and use the same test set eventually.
$\begingroup$
$\endgroup$
asked Dec 22, 2015 at 11:17
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
Briefly,
if you write a
tuned_modelfunction that internally does its own cross validation on the supplied training data and returns the tuned model, you can use this function just analogously to the model training function that provides the other model that does not require any tuning.You are right that you want to do a pairwise comparison, e.g. by McNemar's test.
Before doing any of this, find out what sample size you need in order to get meaningful comparison results for the performances you expect of both models.
I.e. carefully design your comparison study.Alternatively as a quick sanity check, you can plug best-case (and worst-case) scenario data into the pairwise test and have a look whether you have any chance at all to achieve results that can lead to meaningful interpretation in your application framework.
Update: note that this concept of "splitting" allows more sophisticated methods of splitting. You can have a bootstrap or iterated cross validation whenever you do a single split, and if done correctly, they are well worth the effort.
answered Dec 22, 2015 at 12:35
-
$\begingroup$ Thank you for the fast reply. I will definitely consider the McNemar's test! Can I ask you something related to my problem? I don't know how familiar you are eith rpart but I just ask you. When splitting my data into train and test set and then use rpart to estimate a model based on my training set, this basically levaes me with train, test and validation set. Since rpart is using 10-fold CV on my training set. Is this correct? $\endgroup$ Commented Dec 22, 2015 at 13:15
-
1$\begingroup$ Patrick, I've never used single trees, only random forests as single tree predictions are far too unstable with my data. Also, McNemar is only one example. It does work for class label output, but as Frank Harrel rightly points out, if your model is able to produce scores rather than the class that's much preferable. You'll pay for the conceptual ease of right/wrong (accuracy-like) comparisons based on class label predictions by low power or the necessity for huge sample sizes. $\endgroup$ Commented Dec 23, 2015 at 22:56
$\begingroup$
$\endgroup$
Some overall points to consider. First, classifiers are seldom as valuable as predictors. Second, make sure you use a proper accuracy scoring rule. Third, data splitting is highly inefficient and arbitrary; consider bootstrapping. Much has been written about these issues on this site. Please research the site.
answered Dec 22, 2015 at 13:26
$\begingroup$
$\endgroup$
It is NOT correct to
use the train set to estimate one of the classifiers and only 80% of the train set to estimate the other
as you're introducing additional confound and your comparison will not be singularly focused on the abilities of the two classifiers.
As in any proper comparison, you would want to control essentially everything else to be identical and vary one parameter or technique of interest, to be able to compare the effects (change in accuracy, AUC etc) of such a change.
