I'm trying to compare two classifiers using cross-validation. For one of the classifiers I'd like to use a validation set to tune a parameter. However, for the other one, I just want to use the training set and no validation set. Writing my own cross-validation loop, I'm not sure how to implement this. Is it correct to use the train set to estimate one of the classifiers and only 80% of the train set to estimate the other? This way, I could then tune one of the models and use the same test set eventually.
if you write a
tuned_modelfunction that internally does its own cross validation on the supplied training data and returns the tuned model, you can use this function just analogously to the model training function that provides the other model that does not require any tuning.
You are right that you want to do a pairwise comparison, e.g. by McNemar's test.
Before doing any of this, find out what sample size you need in order to get meaningful comparison results for the performances you expect of both models.
I.e. carefully design your comparison study.
Alternatively as a quick sanity check, you can plug best-case (and worst-case) scenario data into the pairwise test and have a look whether you have any chance at all to achieve results that can lead to meaningful interpretation in your application framework.
Update: note that this concept of "splitting" allows more sophisticated methods of splitting. You can have a bootstrap or iterated cross validation whenever you do a single split, and if done correctly, they are well worth the effort.
Some overall points to consider. First, classifiers are seldom as valuable as predictors. Second, make sure you use a proper accuracy scoring rule. Third, data splitting is highly inefficient and arbitrary; consider bootstrapping. Much has been written about these issues on this site. Please research the site.
It is NOT correct to
use the train set to estimate one of the classifiers and only 80% of the train set to estimate the other
as you're introducing additional confound and your comparison will not be singularly focused on the abilities of the two classifiers.
As in any proper comparison, you would want to control essentially everything else to be identical and vary one parameter or technique of interest, to be able to compare the effects (change in accuracy, AUC etc) of such a change.