Random search is one possibility for hyperparameter optimization in machine learning. I have applied random search to search for the best hyperparameters of a SVM classifier with a RBF kernel. Additional to the continuous Cost and gamma parameter, I have one discrete parameter and also an equality constraint over some parameters.

Now, I would like to develop random search further, e.g. through adaptive random search. That means for example adaptation of the search direction or of the search range.

Does somebody have an idea how this can be done or could reference to some existing work on this? Other ideas for improving random search are also welcome.


closed as unclear what you're asking by Sycorax, Greenparker, whuber May 24 '16 at 11:54

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    $\begingroup$ Many of the methods described here are applicable to this problem. stats.stackexchange.com/questions/193306/… $\endgroup$ – Sycorax May 24 '16 at 1:29
  • $\begingroup$ Could you elaborate on what needs "improving"? Are you looking for methods that are faster, more accurate, more general, more efficient in other computing resources, parallelizable, more elegant, specifically suitable for your particular circumstances, or something else? $\endgroup$ – whuber May 24 '16 at 11:54
  • $\begingroup$ @whuber Sorry for the unclarity. With improving I mean reducing the number of iterations needed until convergence and if possible more elegant. $\endgroup$ – MachineLearner May 24 '16 at 15:23
  • $\begingroup$ Thank you. Since that's precisely what you do when the cost of each evaluation is high, it looks like the link provided by @GeneralAbrial goes to a duplicate of your question, where you can find several good answers and commentary. $\endgroup$ – whuber May 24 '16 at 15:52

Global optimization methods may be what you are looking for.

  • DIRECT (Jones et al., 1993) is a relatively simple yet working gradient-free global optimization algorithm. It basically divide the search space into sub-rectangles during its search, so that's why it's named DIviding RECTangles.
  • Bayesian Optimization (See Shariari et al., 2015 for the comprehensive review) is more sophisticated than DIRECT, and is a topic of intense research in the machine learning field.

These algorithms are not direct descendents of the random search. However, they are "random" in a sense that they sometimes do exploration (instead of exploitation). They are "adaptive" in a sense that they do exploitation when a subregion in the search space looks promising.


Not the answer you're looking for? Browse other questions tagged or ask your own question.