7
$\begingroup$

In machine learning every algorithm has a set of hyperparameters which needs to be optimized for best prediction performance. The simplest method for this optimization is called grid search which means to try all possible parameter value combinations. In this way one can find the best parameter values.

However, this does not give any insight about the interaction of parameter values. For example suppose that we have 5 hyperparameters p1, p2, p3, p4, and p5. There might be these kind of facts: as the value of p1 increases given that p3's value is low prediction performance increases. However, if p3's value is high then p1's value has no effect. There might be many more interesting facts like these. Is there any method for finding these kind of facts?

|cite|improve this question
$\endgroup$
4
$\begingroup$

Many dedicated optimization methods exist for hyperparameter tuning. Sequential model based optimization (a Bayesian inspired method) is a particularly popular research topic, for instance here. Metaheuristic approaches like genetic algorithms, particle swarm optimization and simulated annealing are also common, see for instance here.

If you want to model the effect of hyperparameters, random search is a good sampling strategy to start from.

You can find implementations of such optimization methods in tuning libraries like Optunity, HyperOpt and Spearmint.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I know that such optimization methods exist. But my problem is not to find the best parameter values, I want to understand the interaction among these parameters. $\endgroup$ – Mario Feb 5 '15 at 20:14
  • $\begingroup$ Have a look at the random search paper I mentioned. You can use your favorite regression approach to model the (nonlinear) effect of hyperparameters. You can use this to postprocess the traces of pretty much any solver. Just don't use grid search to explore the search space, because that's very bad for subsequent modeling. $\endgroup$ – Marc Claesen Feb 5 '15 at 20:22
0
$\begingroup$

Design of Experiments on Wikipedia might be a good place for additional reading. At its most general, it covers how to make progress in problems where there is so much complexity that the intuitive approach is no longer constructive.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.