Common question: What are the different options (in common languages like R or Python) available for optimizing hyperparameters? I am primarily interested in implementations in R that can work with XGBoost.
My question has been asked before, but I didn't see any recent revisits to this question. Thorough responses can be seen here.
Based off my search, the most common methods are
- Grid search, which is inefficient and can often fail to optimize
- Random search - more efficient than grid search. see.
- Bayesian optimization - Implemented in R with rBayesianOptimization and MlBayesOpt
- Particle Swarm Optimization. Implemented in psoptim in R.
Past those, what other algorithms are implemented in R? One of the linked questions mentions LIPO for example, but I couldn't find any R package.
As of June 2018, what options we do have?