3
$\begingroup$

I would like to do some basic check that my huge supervised ML model is not over-fitting (two-class problem). I was thinking about the following:

  1. Randomly permute labels using entire dataset
  2. Re-do learning, collect performance metrics
  3. Repeat until some number of iterations

Is this a good way to go? Step two involves quite heavy computation: cross-validation, building & aggregating models, etc.

$\endgroup$
2
  • 1
    $\begingroup$ Why would this check over fitting? Unless I'm confused about what you're doing, this would only check whether your model was better than random chance. Cross validation is generally used for checking whether your model is over fitting. $\endgroup$
    – Cliff AB
    Dec 30, 2015 at 16:28
  • $\begingroup$ Maybe you could be interested in looking at "Permutation tests for studying classifier performance" (2010) by Ojala & Garriga, jmlr.org/papers/volume11/ojala10a/ojala10a.pdf $\endgroup$ Dec 30, 2015 at 19:50

1 Answer 1

1
$\begingroup$

This makes some sense. If the model is (over)fitting to coincidences in the data instead of the true pattern, then it is plausible that it will (overfit) to coincidence when you permute the labels.

However, if you are willing to be so heavy on computing and fit your model over and over, alternatives come to mind. A notable one is to bootstrap the data and redo the entire modeling process on the bootstrap sample before evaluating on the entire sample. I discuss this process here. Another possibility is to use some kind of cross-validation where you fit a model to a subset of the data and then test on the remaining data. Then you repeat this for many different sets of training data with holdout data for testing. An advantage of bootstrap is that you use the entire data set for developing your model, rather than decreasing the sample size by sacrificing precious data to the holdout set.

Neither bootstrap nor cross-validation seems inherently more computationally demanding than your proposed ideas to permute the labels.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.