3
$\begingroup$

I mainly use k-fold cross-validation for parameter tuning and model selection for prediction problems. Now, is there a standard or if not a less-known way to measure the sensitivity of the parameters over the cross-validation error, and to quantify it? Also, is it standard to look at the variability of the cross-validated error across folds as well? I know that the question around the variability has a lot of caveats, but what is a good way to do this, say if I was reporting cross-validation results and the cv errors were either close by/ were relatively changing based on the size of the scaled dataset- then how can I report the variability apart from the cv errors itself?

Note: Assume that I have just two parameters to tune, and I use a discrete grid of the two parameter combinations

$\endgroup$
2
  • $\begingroup$ You may already be doing this, but since you're on a 2d grid, it's easy to look at a heatmap or contour plot of performance over the parameter values. Those are definitely reportable as well and can give you a good sense of how sensitive your model is to parameter selection. (I don't know of a good quantitative summary of that sensitivity, though.) In terms of variability across folds: sometimes people look at standard deviations or something of that. Probably more useful is to, say, instead of doing 10-fold CV, do several runs of 2-fold and look at variations across that. $\endgroup$
    – Danica
    Commented May 31, 2013 at 20:04
  • $\begingroup$ @Dougal Likewise, I have had these thoughts as well ;) and was looking for something more definitive-on this problem $\endgroup$
    – hearse
    Commented Jun 1, 2013 at 22:07

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.