Hyperparameter tuning on the training data? Cross validation I have a set of data (around 1500 data points) with 75 parameters and I am trying to compare the performance of SVM, Decision Tree and a few other supervised techniques.
My data set is not perfectly balanced (900 - Cat 1 vs 600 - Cat 2), so I decided to run SMOTE method to balance it. Here is what I did.

*

*I split the data 80% to 20%

*Used SMOTE on the training data

*Then, used grid-search to optimize my parameters on the training data

*Then, tested & compared the models using the testing data

I was told that due to my data size, I should have used cross validation method. Also, I was told that hyperparameter optimizations should have been done with a validation division.
Here is what I think but I may be wrong (I decided not to use SMOTE anymore since the date set is not terribly imbalanced.

*

*Use grid-search to optimize hyperparameters.

*Use cross-validation to generate results

*Then, compare methods (AUC, F1 e.g.)

I have a feeling that this is not right. Should I have a validation fold/file and tune hyperparameters there?
 A: It is not so clear to me how you want to split your data in your second approach.
Maybe I can suggest the following:

*

*Uee k-fold cross validation for hyperparameter tuning. You take your data set and split 80-20 (90-10 or 80-30 would also work). The 80% you use for tuning using cross-validation. So you split your 80% in k many train-validation splits and try different parameters (Gridsearch would be one way). You take the average loss for these k validation sets for all different parameters and the setting with the lowest average loss value would be your wining set of parameters.

*Take your 20% (or whatever you decided on in step 1) and run your best parameters on this test set. This is the error you document as your final model loss. Because this is the closest to your expected value on how your model would generalise for a new unseen set of data. At least in the framework of empirical risk minimisation.

By the way: in your final test error you hope to be somehow close to the one from your validation error. Otherwise you might have overfitted in the tuning process. If this is the case, you might need to start the tuning process again.
A: What you were told is correct, i.e.:

*

*If you tune parameters, you have to do an inner / outer split, tune on the inner and evaluate the performance of the final model on the outer split

*If you have the computing power, CV is less wasteful than a simple validation. However, note that with an inner / outer CV, computing time goes as k1*k2, where k1 is the number inner and k2 the number of outer folds, so in many cases people choose at least for the outer split a simple validation. However, in principle, using CV for both splits will make best use of your limited data.

