parameter tuning using nested cross validation

Question

Parameter tuning in SVM has been performed using a nested cross-validation(CV) approach with 45 folds(outer loop) and 13 folds(inner loop). In this process, the outer loop will have 45 prediction tasks with a chosen best value for C from the inner loop. I have taken the values 0.5, 5, 50, 500 for tuning the parameter C and observed that F1-score was highest 34 times for value 5, 6 times for value 50, 5 times for value 500.

Would it be a good idea to test the model on an external dataset(not involved in nested CV) with value 5 for C(highest frequent best value) ?

Answer 1

Getting a final performance estimate for the best model parameters found with cross validation (CV) on new data is a good idea. As you suggest, this should be be done using a dataset left out during CV that is yet unseen by the model.

Having said this, you could combine your inner and outer CV for doing model tuning and model selection at the same time, e.g. using repeated cross validation. For example, if you used 10 partitions with 20 repeats (including re-partitioning), this would leave you with 200 CV performance estimates per model type and parameter set to evaluate, from which you could decide upon the best suited model type and parametrization at the same time. Have a look at this question for some more details and an example on how to do this with e.g. R caret.

PS: as you are only tuning C I'm assuming you are using a linear kernel?

Edit: this answer on a related question dealing with nested cross validation has excellent information on how to use repeated cross validation for model tuning and model selection and what its benefits are.