6
$\begingroup$

When conducting a grid search over a range of parameters of a predictive model which is itself subject to randomness (such as a random forest with bagged features), should you set a seed for the predictive model, so that for each round of the grid search, the model is initialized the same? It seems straightforward, that you ONLY want to test the parameters, and the less variance, the better, but is there any scientific consensus on this?

$\endgroup$
2
  • $\begingroup$ To give a concrete example, if you're comparing the performance of RandomForest with max_features=2 or max_features=10, are you asking if it would be in some sense better to start each test with the random number generator initialized to the same seed? $\endgroup$ Commented Jan 18, 2018 at 20:33
  • $\begingroup$ Yes, exactly, but it might be that it doesn't have a significant effect, depending on how the rng works $\endgroup$
    – Sam
    Commented Jan 19, 2018 at 9:35

4 Answers 4

5
$\begingroup$

The premise that the same random seed will lead two randomized algorithms to have more similar performance is extremely dubious (except perhaps for the most similar and specially structured of algorithms over the smallest of samples).

An analogy

Using a Monte-Carlo simulation, let's say you're trying to estimate a casino's house take in:

  • Game A: blackjack where the dealer hits on soft 17
  • Game B: blackjack where the dealer stands on soft 17

Would it make the comparison less noisy if Game A and Game B used the same order of cards (i.e. started with the same random seed)?

No! (Not in any meaningful way.) The moment Game A leads the dealer to take an additional card (compared to Game B), the games are no longer in sync: players will be dealt different hands, cards that would have gone to the dealer instead go to a player etc.... Just one card offset makes a huge difference, and everything will just diverge from there.

There may be some special case algorithms where the small differences don't just compound, but I would think these are unusual cases.

$\endgroup$
1
  • 2
    $\begingroup$ I'm afraid that your analogy may not be perfectly clear for those of us who weren't yet in Vegas ;) $\endgroup$
    – Tim
    Commented Jan 18, 2018 at 21:06
1
$\begingroup$

That is an ongoing reseach topic (hyperparameter optimization). A very popular technique following the idea you formulate in your question is random search.

Once you see it, the idea is quite simple, and it is shown to work well in practice. Consider you search space with a finite maximum. Take the 5% interval around the maximum. If you sample at random, you have 5% chance of landing in that interval. The probability that when you sample n times you hit at least one time this interval is, $$ 1-(1-0.05)^n $$

Now, you need to determine, how many times you need to sample. First you set a threshold for that probability. For example, you want to have a 95% confidence that you will eventually hit the interval once. Then it yields that n has to be at least 60.

$\endgroup$
1
  • 3
    $\begingroup$ I don't think this answers the question that was asked. $\endgroup$
    – Tim
    Commented Jan 19, 2018 at 15:44
1
$\begingroup$

It seems straightforward, that you ONLY want to test the parameters, and the less variance, the better

Well, it isn't that straightforward. @MatthewGunn already explained that it typically won't help as you apparently thought.

In general, if you encounter variance there are two quite opposide strategies of dealing with it.

  1. (As you suggest,) reduce it as much as possible by restricting experimental conditions, or
  2. measure and account for it.

Strategy 1 will only help if the restricted conditions can be applied to your further process. If they can not, you need to go with approach 2.

Besides, there are further points to consider:

  • If you fix the seed and for the algorithm you tune that does indeed lower the variance, then your conclusions are limited to that one seed (always the case with strategy 1). This may or may not acceptable for the task at hand - but you need to consider this point carefully.

  • Besides algorithms where the seed has negligible influence on the randomness (see @MattewGunn's answer) there are also algorithms where the combination of seed and training cases together lead to variance. In that situation, you may be successfully suppressing variance during the grid search using the same training set and the same seed but e.g. a final model built form the larger base of (training + optimization test cases) for training will be subject to additional variance. In other words, your grid search overfits.

  • Strategy 2 becomes particularly important, if some parameter of your grid may be influencing this variance: you'd then be able to consider whether the apparent optimum in your grid may be due to variance.

$\endgroup$
1
$\begingroup$

It seems that there is no scientific consensus on it.

I think that Monte Carlo analogy by @Matthew is not perfect. E.g. if you had a neural network and you did a grid search of learning rate and momentum, then using the same random seed would lead to the same initialization which seems a good idea, so I agree with your intuition.

Key point here is different. If variance coming from random seed is significant compared to variance coming from different choice of hyper-parameter, then grid search may not have sens. For different seeds it may find different optimal hyper-points.

If such a case occurs, you may want to perform repeated cross validation, for more see this site enter image description here

So the final answer is: if fixing seed matters, something is going wrong.

$\endgroup$
1
  • $\begingroup$ That's a good point that the random seed may matter more if you're minimizing non-convex functions (have problems with local minima) and your initial condition $\boldsymbol{\theta}_0$ is a random draw (that will become the same for each grid point if you fix the seed). I also agree that if substantive results are sensitive to fixing the seed, that's a problem. It implies results are not robust. $\endgroup$ Commented Feb 8, 2018 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.