2
$\begingroup$

What is the correct strategy of using Grid Search? Am I understand correctly that to use correctly Grid Search I should:

  1. Give Grid Search initial parameters that have wide range. For example if parameter alpha usually ranges from 0 to 100 I should give list of parameters like [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100] and if parameter beta usually ranges from 0.1 to 0.0001 give list of parameters [0.1, 0.01, 0.001, 0.0001]?

  2. Now we picked the parameters 'alpha': [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100], 'beta':[0.1, 0.01, 0.001, 0.0001] and we run Grid Search with them.

  3. We got best parameters from Grid Search: 'alpha': 50, 'beta':0.0001

What is the next step here? How should we run Grid Search again with:

Variant 1:

'alpha': [0, 10, 20, 30, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 70, 80, 90, 100], 'beta':[0.1, 0.01, 0.001, 0.0001, 0.00001, 0000001]

OR

Variant 2

 'alpha': [42, 44, 46, 48, 50, 52, 54, 56, 58], 'beta':[0.0001, 0.00001, 0000001]

Is variant 2 good variant? I thought that variant 1 is better because the best combination of parameters may be 'alpha': 30, 'beta':0.00001. Is it possible? Or I should remove all parameters that does not perform well in first Grid Search as in variant 2?

  1. Repeat step 3 until found parameters that are giving best score.

So my question is:

  1. Is choosing parameters in step 1 correct way of doing that?
  2. Which variant of step 3 should I use?
  3. How does that strategy looks overall? Is it good strategy or anything should be changed?
$\endgroup$
1

2 Answers 2

1
$\begingroup$

You are right that the best combination may be 'alpha': 30, 'beta':0.00001 but the best combination might be always beyond what you are exploring. The point is to establish a big enough interval to make sure you are capturing the best combination, and for that you use a coarse-enough discretization to explore it efficiently. After that, you increase the dicretization around a narrower interval, so the computational cost is the same as in the previous step, and you can find the optimum combination with a higher precision. Thus, variant 2 is more sensible. You can repeat the process as long as it is worth it (at some point the accuracy will not improve appreciably).

In general, it's a good strategy if you use variant 2, but is also the simplest one. The first thing you should do (if you are not doing it now) is to combine it with cross-validation. You could go beyond grid search by using e.g. gradient boosted based techniques which explore the parameter space automatically, following a path that leads toward optimum parameters much more efficiently that a grid search, but depending on your purpose, this might be excessive.

$\endgroup$
7
  • $\begingroup$ Thank you. By gradient boosted techniques you mean XGboost algorithm or there is some Grid Search alternative for all algorithms? $\endgroup$
    – vasili111
    Commented Feb 21, 2021 at 1:18
  • $\begingroup$ XGboost is one library implementing gradient boost (maybe the most famous). The implementation of the algorithm can be found in XGboost or in other software. I don't fully understand the question "Grid Search alternative for all algorithms?" If by algorithm you refer to ML model, then you can apply it to any model in order to optimize its parameters. One thing is the model, and a separated one is the optimization strategy. $\endgroup$
    – rasmodius
    Commented Feb 21, 2021 at 1:23
  • $\begingroup$ Thank you. Your explanation is very useful. I thought before that XGboost was some kind of improved version of Random Forest only. So, am I understand correctly that gradient boost (or XGboost in particular) can be used to find optimal parameters for SVM, RF, ElasticNET and many others? $\endgroup$
    – vasili111
    Commented Feb 21, 2021 at 1:42
  • $\begingroup$ Also some people use Grid Search to find optimal parameters for XGboost, like here: kaggle.com/phunter/xgboost-with-gridsearchcv $\endgroup$
    – vasili111
    Commented Feb 21, 2021 at 1:47
  • $\begingroup$ I should be more precise. You can use gradient-based algorithms to explore the parameter space, like gradient descent. But gradient boost refers to a specific kind of ML model, as you said related to trees. Sorry, I mixed the names. You can apply gradient descent to any model, but some models have parameters that are bounded so they can't exceed some limits, or can take only discontinuous values. $\endgroup$
    – rasmodius
    Commented Feb 21, 2021 at 2:00
1
$\begingroup$

There is no absolute correct strategy, and there are resource trade-offs. Surely, you can't try everything. Variant 2 performs a local search around the previous optimum, and is a typical strategy. It researches for the optimum in a fine tuned grid.

If you also want to be a bit explorative, you an try an approach such as variant 1. This may result in finding other local optima, such as the pair (alpha=30, beta=0.00001) you mentioned. However, in variant 1, you're repeating some/many calculations, and you should omit the hyperparameter pairs that have been already tried in the previous iteration.

I'm afraid there is no definitive answer for your first question. Options to try heavily depend on the meaning of the hyperparameter and the possible common values of it. If you think they're all equally likely, you can surely choose something like [0,10...100].

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.