Common question: What are the different options (in common languages like R or Python) available for optimizing hyperparameters? I am primarily interested in implementations in R that can work with XGBoost.

My question has been asked before, but I didn't see any recent revisits to this question. Thorough responses can be seen here.

Based off my search, the most common methods are

  1. Grid search, which is inefficient and can often fail to optimize
  2. Random search - more efficient than grid search. see.
  3. Bayesian optimization - Implemented in R with rBayesianOptimization and MlBayesOpt
  4. Particle Swarm Optimization. Implemented in psoptim in R.

Past those, what other algorithms are implemented in R? One of the linked questions mentions LIPO for example, but I couldn't find any R package.

As of June 2018, what options we do have?

  • $\begingroup$ I don't think the landscape has changed that much since I last updated my answer (in Feb.). $\endgroup$ – Sycorax Jun 13 '18 at 21:09

It's all about black-box optimization. Especially, we are most of the time facing costly black-box functions.

Here are slides of a course from l'Ecole Polytechnique (best French engineering school) about black-box optimization. It mentions:

  • $\begingroup$ I'm not sure how much of this is "new". The Jones paper appears in the previous thread. Kriging pre-dates Jones by more than 25 years. $\endgroup$ – Sycorax Jun 13 '18 at 23:04
  • $\begingroup$ Well indeed it's not "new" if it is not a paper from 2017-2018. But Kriging is still an active topic of research. I mentioned those methods because they were not in the 4 points but were cited in a notable course. $\endgroup$ – Arius Jun 14 '18 at 6:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.