I am reading about the hyperparameters optimization for a machine learning model. I am reading "Practical Recommendations for Gradient-Based Training of Deep Architectures" paper. Author speaks about manual search, grid search and random search. What is the difference between Manual serach and grid search. What I understand is in both cases, we define a zone of interest and we select values from it to test our model on validation set. The best values of hyperparameters are choosen by minimizing a criteria, for example, error classification on validation set. May be in a grid search approach, we try more values! Any explanation?
$\begingroup$
$\endgroup$
1
asked Jan 9, 2017 at 16:27
-
$\begingroup$ You should provide a full citation and/or link for the paper you mention. Going by the apparently similar descriptions in section 11.4 of Ian Goodfellow's Deep Learning Book, my impression is that in grid search a regular grid is laid out in parameter space, and all values are tried, then the global optimum is chosen. On the other hand, manual search is not an exhaustive search, and the parameter values are adjusted based on educated guesses, so a "trial and error" approach. $\endgroup$– GeoMatt22Commented Jan 9, 2017 at 16:46
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The difference isn't especially enlightening. Grid-search pre-specifies some set of tuples up front and tries all of them. In manual search, a human adjusts the parameters, possibly incorporating knowledge about how those adjustments will influence the behavior of the model and estimation procedure.