# Hyper-parameter optimization via random search

I’m working on a classification system which consists of an auto-encoder for feature learning and logistic regression for classification. The system has five hyper-parameters as enumerated below.

1. Number of features it's learning via auto-encoder
2. Weight decaying parameter of the auto-encoder
3. Weight decaying parameter of the logistic regression
4. Sparsity parameter of the auto-encoder
5. The weight of the sparsity penalty term of the auto-encode

I’m planning to use random search for obtaining optimum values for these parameters. More information about the random search for hyper-parameter optimization can be found in this paper

My question is, in order to perform a random search we need to identify the appropriate ranges for each hyper-parameter. For example, the weight decaying parameter of the auto-encoder belongs to [X, Y].

So do you know a published paper to extract these initial values?

• +1 for the link to that excellent paper. My guess is that a definitive answer to your question would be difficult to formulate, it will depend on the data. This is why we need to search in the first place. One shortcut I use is to grab a small subset of the data, divide into train and test sets and then use Nelder-Mead to optimise the test error. This at least tells you the order of magnitude of your parameters. If you repeat this several times the spread of the resulting values also give a hint to the reasonably search range. This doesn't also work, depending on the data and algorithm. – Bogdanovist Jul 25 '12 at 5:15
• @Bogdanovist I recommend shying away from Nelder-Mead. It gets stuck almost surely when optimizing hyperparameters as they do not meet the assumptions at all. Evolutionary approaches (such as particle swarms or CMA-ES) and model based methods (such as EGO) are far better for tuning. Implementations of commonly used methods are available in the Optunity library (currently usable in Python and MATLAB). – Marc Claesen Aug 21 '14 at 10:27

For some of these parameters you can simply reason about what the range should be. For example, since the sparsity parameter is the desired average activation of the hidden units, this will only make sense if it is in $(0, 1)$, assuming you're using a sigmoid activation. However, you can still narrow this down farther as a sparsity $> 0.5$ is somewhat nonsensical. For other parameters your can look for the values people commonly report in the literature, use this as a rough guess, and expand your ranges if you find it necessary. For example I would probably start with $(0, 0.2)$ for the learning rate.