-1
$\begingroup$

Random search is one possibility for hyperparameter optimization in machine learning. I have applied random search to search for the best hyperparameters of a SVM classifier with a RBF kernel. Additional to the continuous Cost and gamma parameter, I have one discrete parameter and also an equality constraint over some parameters.

Now, I would like to develop random search further, e.g. through adaptive random search. That means for example adaptation of the search direction or of the search range.

Does somebody have an idea how this can be done or could reference to some existing work on this? Other ideas for improving random search are also welcome.

$\endgroup$
4
  • 2
    $\begingroup$ Many of the methods described here are applicable to this problem. stats.stackexchange.com/questions/193306/… $\endgroup$
    – Sycorax
    May 24, 2016 at 1:29
  • $\begingroup$ Could you elaborate on what needs "improving"? Are you looking for methods that are faster, more accurate, more general, more efficient in other computing resources, parallelizable, more elegant, specifically suitable for your particular circumstances, or something else? $\endgroup$
    – whuber
    May 24, 2016 at 11:54
  • $\begingroup$ @whuber Sorry for the unclarity. With improving I mean reducing the number of iterations needed until convergence and if possible more elegant. $\endgroup$ May 24, 2016 at 15:23
  • $\begingroup$ Thank you. Since that's precisely what you do when the cost of each evaluation is high, it looks like the link provided by @GeneralAbrial goes to a duplicate of your question, where you can find several good answers and commentary. $\endgroup$
    – whuber
    May 24, 2016 at 15:52

1 Answer 1

1
$\begingroup$

Global optimization methods may be what you are looking for.

  • DIRECT (Jones et al., 1993) is a relatively simple yet working gradient-free global optimization algorithm. It basically divide the search space into sub-rectangles during its search, so that's why it's named DIviding RECTangles.
  • Bayesian Optimization (See Shariari et al., 2015 for the comprehensive review) is more sophisticated than DIRECT, and is a topic of intense research in the machine learning field.

These algorithms are not direct descendents of the random search. However, they are "random" in a sense that they sometimes do exploration (instead of exploitation). They are "adaptive" in a sense that they do exploitation when a subregion in the search space looks promising.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.