Should we always do CV?

Question

My question: should I do CV even for a relatively large data set?

I have a relatively large data set and I will apply a machine learning algorithm to the data set.

Since my PC is not fast, the CV (and grid search) takes sometimes too long time. In particular a SVM never ends because of lots of tuning parameters. Thus if I do a CV, then I need to pick a relatively small data.

On the other hand the validation set should be also large, so I think that it is a good idea to use a validation set which has the same (or larger) size as the training set. (Namely instead of CV I use a large validation set for parameter tuning.)

So I have now at least two options.

1. do CV on a small data set.
2. use relatively large training set and validation set without CV.
3. other idea.

What is the best idea? Theoretical or practical opinions are both welcome.

Answer 1

In general, you don't have to use cross validation all the time. Point of CV is to get more stable estimate of generalizability of your classifier that you would get using only one test set. You don't have to use CV if your data set is enormous, so adding data to your training set won't improve your model much, and few missclassifications in your test set just by random chance, won't really change your performance metric.

By having a small training set and a big test set, your estimation will be biased. So it will be probably worse than what you would get using more training data and optimal hyperparameters that you found might be different for bigger dataset, simply because more data will require less regularization.

However, getting optimal hyperparamters is not the important part anyway and it won't improve the performance dramatically. You should focus your energy to understand the problem, creating good features and getting data to good shape.

Here are few things you can consider to speed things up:

1. Train it with fewer features. Use feature selection and/or dimensionality reduction to decrease the size of your problem
2. Use precached kernel for SVM
3. Use algorithms that doesn't need to select hyper parameters in a grid. Especially linear ones like logistic regression with ridge/lasso/elastic net penalty or even linear SVM. Depending on implementation, those classifiers can fit models for all hyperparameters in selected path for the cost of fitting only one
4. use faster implementation for your type of problem (you will have to google it)

and even with slower computer, you can:

5. Use more cores
6. Use GPU

Answer 2

Cross-validation is a tool to estimate the variance of your performance metric due to randomness in the data (and maybe in the learning algorithm if it not deterministic).

So if you use only one split, e.g. 80% train + 20% test and report your performance metric from this single experiment there are good chances that anyone trying to reproduce your experiment using exactly the same parameters will find a different performance figure (sometimes very different). Unless of course you provide the same exact split which is meaningless.

To come back to your question I think you should definitely use CV to report your performance (e.g. do a 10 folds CV and report the mean and standard deviation of the performance metric). Now for tuning your algorithm you may use a much smaller validation set sampled from the training set (make sure it is not included in the test set).

If you're afraid that you won't find the best hyperparameters using a small set then you're probably overfitting your algorithm to the specifics of the dataset. If you can't find a configuration using a small sample that gives a reasonable performance among all folds then the algorithm is probably not very useful in practice.

Also keep in mind some algorithms are simply too slow / don't scale well in some configurations. This is also a part of practical model selection.

Since you mention SVMs, of course most implementations will be slow when trying to find parameters for non-linear kernels by grid search. Grid search has exponential complexity, so use it with very few parameters. Also keep in mind that most libraries provide sensible default parameters (or at least you set one parameter and there are heuristics to set the others).