0
$\begingroup$

If I understand things correctly, in a nested cross validation the inner cross validation is for optimising over the search space of hyperparams, and the outer loop is validating the accuracy the optimal hyperparams determined by the inner loop. i.e.

  • outer cross validation
    • hyperparameter search
      • inner cross validation

We then have the choice of either reusing the same inner cross validation splits (i.e. the same random seed) or we can randomise (i.e. change the random seed) for each inner cross validation of every hyperparam vector candidate we investigate.

On the one hand, I can see that by keeping the seed the same we are only changing one variable (namely the chosen vector of hyperparam candidates), which makes the hyperparam optimisation easier. i.e. if both the hyperparams and data are changing, optimisation is having to optimise over more free variables.

On the other hand, if we randomise the folds for each hyperparam vector candidate there is less chance we will find a local minima / maxima due to the chance of an "unlucky" single choice of inner cross validation split producing a model that is optimal for that single split, but not for other possible splits.

How does the choice of randomise versus not randomise for the inner cross validation affect the hyperparameter vector search optimisation?

I suspect that the answer is very dependent on the size of the hyperparam search space (i.e. the difficulty of optimising the hyperparams) versus the distribution of the data (i.e. the probability of choosing a really "bad" split and wrongly concluding we've found the best vector of hyperparams).

|cite|improve this question
$\endgroup$
1
$\begingroup$

If you search for a zillion hyperparameter combinations, you will start to overfit whatever you're testing those against.

Therefore, I'd be tempted to take one single train/validate split, do your hyperparameter search on that. And then, and only then, evaluate it against some other split or fold or test/validation data set.

Using random splits for each set of parameters sounds like something to avoid because:

  • as you say, for some hyperparameter options, you'll 'get lucky', jsut because of the train/validate split giving a randomly high score
  • you're basically overfitting to every possible train/validate split of your data, leaving you no novel splits to validate against
|cite|improve this answer
$\endgroup$
  • $\begingroup$ The outer cross validation should prevent overfitting. There is no leakage from the inner hyperparameter optimisation into the outer cross validation. As the inner cross validations are independently randomised and therefore independent of the candidate hyperparameters the randomisation would then generate (random, possibly biased) noise in the hyperparameter optimisation. I'm guessing this noise probably doesn't greatly affect the optimisation, as it is likely the vast majority of inner cross validation splits are representative of the entire training data set for a large enough data set. $\endgroup$ – Gary Mulder Apr 2 '17 at 14:13
  • $\begingroup$ You're running hyperparameter optimization for each of the outer splits right? That means you are trying a bunch of hyperparameters against a whole bunch of splits. As far as "for a large enough data set", such a thing is rare in practice ;-) If you have infinite data, you are in a good position :-) Otherwise, generally speaking, one increases the capacity of one's model to match the amount of data available, and the overfitting problem comes back. $\endgroup$ – Hugh Perkins Apr 2 '17 at 14:58
  • $\begingroup$ (If I'm misunderstanding your two loops, perhaps you can add some additional detail in the question about how you are running the two loops, exactly what hyperparameters you are trying, and where etc?) $\endgroup$ – Hugh Perkins Apr 2 '17 at 15:00
  • $\begingroup$ I've edited the question to show the ordering of the three nested loops. $\endgroup$ – Gary Mulder Apr 2 '17 at 20:22
  • $\begingroup$ Can I clarify your second bullet point: "every possible train/validate split of your data" as referring to the outer cross validation and not the inner, then? I had assumed you were talking about the inner cross validation. If so, what you're implying is that the hyperparameter search may end up spending more time searching the inner (randomised) cross validation space than the hyperparameter space, i.e. the choice of inner cross validation split may become more important than the choice of hyperparameters for every outer split. $\endgroup$ – Gary Mulder Apr 2 '17 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.