2018 Update - Different Algorithms for Hyperparameter Optimization

Question

Common question: What are the different options (in common languages like R or Python) available for optimizing hyperparameters? I am primarily interested in implementations in R that can work with XGBoost.

My question has been asked before, but I didn't see any recent revisits to this question. Thorough responses can be seen here.

Based off my search, the most common methods are

1. Grid search, which is inefficient and can often fail to optimize
2. Random search - more efficient than grid search. see.
3. Bayesian optimization - Implemented in R with rBayesianOptimization and MlBayesOpt
4. Particle Swarm Optimization. Implemented in psoptim in R.

Past those, what other algorithms are implemented in R? One of the linked questions mentions LIPO for example, but I couldn't find any R package.

As of June 2018, what options we do have?

Answer 1

It's all about black-box optimization. Especially, we are most of the time facing costly black-box functions.

Here are slides of a course from l'Ecole Polytechnique (best French engineering school) about black-box optimization. It mentions:

• Kriging. There is an available package in R.
• Efficient Global Optimization of Expensive Black-Box Functions, D. Jones et al. 1998
• An adaptive radial basis algorithm (ARBF) for expensive black-box global optimization, Holmstrom et al. 2008