3
$\begingroup$

The assignment is to write a simple ML program that trains and predicts on a dataset of our choice. I want to determine the best model for my data. The response is a class (0/1). I wrote code to try different cross-validation methods (validation set, leave-one-out, and k-fold) on multiple models (linear regression, logistic regression, k-nearest neighbors, linear discriminant analysis). Per model, I report the MSE for each cross-validation method and track the lowest one. I then pick the model with the lowest tracked MSE. This is where I think I went wrong. If I am cross-validating multiple models, should I use the same cross-validation method?

$\endgroup$

4 Answers 4

0
+100
$\begingroup$

As you noted, using various cross-validation techniques can produce various results for various models, making it challenging to compare the effectiveness of various models in an objective way. As a result, using the same cross-validation method across multiple models is recommended. In this way, you can make fair comparisons between the various models' outputs.

Also, if your objective is to find the best model, is this strategy effective? According to machine learning's "no-free-lunch theorem," no single model outperforms all others in all circumstances. The best model for your data will be determined by the structure and size of your data, the problem's complexity, the model's parameters, and the computational resources available.

As a result, it is essential to choose the model and cross-validation method that are "the best" for your data and objectives. You could also consider techniques like hyperparameter optimization.

I hope this helps!

$\endgroup$
3
$\begingroup$

If different cross-validation schemes make a statistically significant difference, then your dataset is probably too small to draw any useful conclusion.

Having lots of cross-validation procedures and taking the best result over models and cross-validation methods is a recipe for over-fitting, in this case over-fitting the model selection criterion (by having more than one to choose from). These sorts of biases can be more substantial than might be expected. See my paper (with Mrs Marsupial)

GC Cawley, NLC Talbot, On over-fitting in model selection and subsequent selection bias in performance evaluation, The Journal of Machine Learning Research 11, 2079-2107 (pdf)

This is essentially because performance estimates have variance - if you try it again and again using different samples of data, you will get slightly different results. So some of the difference in apparent performance between cross-validation estimates is due to the sampling variation favouring one method a little more than it does another. So choosing the best estimator can exploit this variation, rather than genuinely indicating a better model, which would be over-fitting in model selection. This may be a substantial problem unless you have lots of data.

Also, as well as variance of the estimator being an issue cross-validation methods can be pessimistically biased. Most classifiers perform better the more training data you have. For leave-one-out CV, you get an almost unbiased as the training set in each fold is almost as large as the full dataset. If you use 5-fold cross-validation, the training set in each fold is only about 80% of the full dataset, so on average the classifier is expected to perform a little worse than it would appear using leave-one-out cross-validation.

If you just want the best model, estimate peformance using some form of bootstrap, and use as many resamples as you can within your computational budget. Better still, use bagging to combine the classifiers from each resampling, and use the out-of-bag estimate to estimate the performance of the bagged model.

$\endgroup$
1
$\begingroup$

There are a few factors that make it difficult to compare results from different validation methods. Two are the size of training set, size of the test set.

In general, models perform a bit better if the training set is larger. For example, leave-one-out cross validation will have a larger dataset than using a 70/20/10 train/validation/test split. So, you would expect to have better performance from leave-one-out cross validation than from a validation set.

If you're using a larger test set, you will have more precision in your estimates of model performance (accuracy, precision, area under ROC, etc). So, if you used train/validation evaluation, you will be testing a model on a smaller group of people than you would for cross validation, so it's more likely that high performance evaluated on a train/validation split is due to chance.

So, you should use the same method on all types of models. Ideally, you would also use the same exact train/test indices across all models you train. (So, don't create new cross validation indices for each model you train and evaluate)

$\endgroup$
1
$\begingroup$

What is the reason that you have different models? What are you doing with the models?

I want to determine the best model for my data.

For determining the best model it doesn't matter whether you did cross validation differently. What is important is that you compare the models with an independent test dataset.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.