Hyper-Parameter Tuning by Grid Search

Question

I am reading this paper by Bergstra et al. and want to make sure I understood correctly the principle of random search. They say:

We describe the hyper-parameter configuration space of our neural network learning algorithm in terms of the distribution that we will use to randomly sample from that con- figuration space. The first hyper-parameter in our configuration is the type of data preprocessing: with equal probability, one of (a) none, (b) normalize (center each feature dimension and divide by its standard deviation), or (c) PCA (after removing dimension-wise means, examples are projected onto principle components of the data whose norms have been divided by their eigenvalues). Part of PCA preprocessing is choosing how many components to keep. We choose a fraction of variance to keep with a uniform distribution between 0.5 and 1.0. There have been several suggestions for how the random weights of a neural network should be initialized (we will look at unsupervised learning pretraining algorithms later in Section 5). We experimented with two distributions and two scaling heuristics. The possible distributions were (a) uniform on (−1,1), and (b) unit normal. The two scaling heuristics were (a) a hyper-parameter multiplier between 0.1 and 10.0 divided by the square root of the number of inputs (LeCun et al., 1998b), and (b) the square root of 6 divided by the square root of the number of inputs plus hidden units (Bengio and Glorot, 2010).

So if I understand correctly:

In "normal" grid search, a range of values is selected for each hyper-parameter and every combination is tested separately. Eg:

Parameter A: [1,2,3] Parameter B: [4,5,6]

Then we'd test combinations (1,4), (1,5), ..., (3,5), (3,6) for a total of 9 (3x3) combinations.

In the case of random search, we define a "preference" for which values to pick, or in other words distributions.

So, in the case of parameter A being discrete, I could define an 80% probability of A=1, a 15% probability of A=2 and a 5% probability of A=3.

If B was continuous, I would define a probability distribution across its range.

Then, I define a number of runs n where at each run parameters are randomly sampled from the distribution. At each run, I keep track of the validation error and then report the parameter combination with the lowest error. (And possibly also the testing error across the parameter combination with the lowest validation error.)

However, am I right in understanding that the choice of how to define the distributions are left up to humans? In other words, random search has as a prerequisite that the developer decides on the distributions and that only once these have been established it can be used to find optimal hyper-parameter configurations?

Answer 1

Yes, the distributions used for random hyper-parameter search are typically left up to the user. For example, consider this real-world snippet of code using scikit-learn's RandomizedSearchCV class:

from sklearn.model_selection import RandomizedSearchCV
from scipy import stats
...
param_dist = {
    'n_estimators': stats.randint(100, 1000),
    'learning_rate': stats.uniform(0.01, 0.07),
    'max_depth': [3, 4, 5, 6, 7, 8, 9],
    'colsample_bytree': stats.uniform(0.5, 0.45),
    'min_child_weight': [1, 2, 3]
}
random_search = RandomizedSearchCV(
    pipeline,
    param_distributions=param_dist, 
    n_iter = 50, 
    scoring = 'neg_mean_square_error', 
    error_score = -99, 
    verbose = 10, 
    n_jobs = 1)

Here I've really only used uniform distributions (both discrete and continuous) but there's absolutely no restriction on passing an arbitrary distribution.

The more mature and very popular hyperopt package takes a similar approach and also supports a large number of standard distributions.