# Common values used for hyperparameter grid search?

When performing a grid search for exploring optimal hyperparameters, what are the typical values, or ranges, that are commonly used for alpha, epsilon, gamma, lambda, C, etc? I'm not focused on a particular algorithm at the moment. And I realize each problem is going to be different. But just wondering if anybody has a resource they can share that provides typical values or ranges that would be good starting points for hyperparameter tuning.

• Hyperparameters of what model? Obviously you would not have the same hyperparameters for totally different models.
– Tim
Commented Jun 4, 2016 at 15:14
• This question seems too broad. I think you could ask for a specific algorithm though. Commented Jun 4, 2016 at 15:19
• Sorry for the general nature of the question. Just thought someone may know of a blog post, book, etc that had some suggestions for some beginning parameter values to use in each of a variety of different models. For example, with ridge regression maybe try alpha = [1, 0.1, 0.01, 0.001, 0.0001, 0]; with SVM try C = [0.0001, 1000, 10000, 100000]; with random forest try minimum samples per leaf = [1, 5, 10, 50, 100, 200, 500]; and so on but with all parameter options for each algorithm (and for most of the commonly used algorithms). Commented Jun 4, 2016 at 17:01

This is going to be very dependent on the algorithm, but I can answer for glmnet.

Glmnet is fit through a cyclical coordinate descent, we apply an update rule to each coordinate in turn and iterate until convergence. The update rule is equation 5 in this paper.

As the authors note, this update rule implies that, if the some parameter $\beta_j$ is zero at some iteration, then it stays zero after the iteration if and only if

$$\left| \langle x_j, y \rangle \right| < N \lambda \alpha$$

where $N$ is the number of training data points. If we let $\tilde \lambda$ denote the minimum lambda for which all parameters are estimated as zero (the set of all such lambdas is clearly an unbounded interval including infinity), the above relation tells us that

$$N \alpha \tilde \lambda = \max_j \left| \langle x_j, y \rangle \right|$$

that is, all parameters are estimated as zero when

$$\lambda > \frac{1}{N \alpha} \max_j \left| \langle x_j, y \rangle \right|$$

This gives us bounds on our grid search

• When $\lambda = 0$ we get the un-regularized model.
• When $\lambda > \tilde \lambda$ all parameters are zero.

To fill in the middle, glmnet breaks up this interval into multiplicatively even subintervals, i.e., subintervals whose endpoints are equally spaced on the logarithmic scale. My assumption is that this was done given some empirical evidence, but I'm not really sure.

Hyperparameters are often searched on a logarithmic scale. This particularly holds for regularization/penalty terms, and cases where the proper scale is unknown. Other hyperparameters can be searched on a linear scale if there's a priori reason to do this. In some sense, you can think of the search range as a 'hyperprior'.

When searching for values of multiple hyperparameters, random search may be better than grid search. The reason is that performance can be more sensitive to some hyperparameters than others, and the set of relevant hyperparameters isn't known ahead of time (it can vary with the dataset). In this case, grid search would waste iterations on adjusting hyperparameters that don't make much difference, while keeping the others fixed. Random search would adjust all hyperparameters simultaneously. Adjustments to hyperparameters that don't matter won't change anything, and adjustments to those that do will have a chance at improving performance.

See this paper:

Bergstra and Bengio (2012). Random Search for Hyper-Parameter Optimization

To give a very general answer to your very general question, I usually start out, at least, by choosing ten or so values that seem to cover the range of the parameter. If the range is [a, b] for some real a and b, the values could be evenly spaced between a and b. If the range is (0, ∞), the values could be 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1, 1e1, 1e2, 1e3, 1e4, 1e5. If the range is (∞, ∞), the values could be -1e5, -1e3, -1e-2, 1e-2, 1e3, 1e5.

Thought I'd post what I've found through additional research in case anyone else stumbles across this looking for the same info. It turns out the caret package for R has baked in standard values/distributions for performing grid and random searches.

To see what values or distributions caret is using as standard, you can view the code for each model available in caret on github:

https://github.com/topepo/caret/tree/master/models/files

When looking through the code, simply look for the "grid" function. The parameter settings within the if(search == "grid") statement would apply to grid searches. The parameter settings within the following else statement would apply to random searches.