A question from a newbie: I played around with parameter tuning (grid, random search) in R (caret, xgboost) and my observation is as follows: in terms of accuracy gains I was able to get 3 - 7% but not more. And that's the limit. Meaning that higher accuracy gains could be reached with new features as well as other models/ensembles. Am I wrong?

If someone could point me to a tutorial for parameter tuning exploration, would be nice.


The addition of features or the introduction of tranformations can significantly increase your model's performance.

Now, given that your dataset is fixed and you want to exploit all the power of your model I would suggest Bayesian Optimization. Bayesian Optimization is a state-of-the-art sequential design strategy for global optimization of black-box functions. Assuming that close input values will have close outputs, we model the observed accuracy values using a flexible Gaussian Process prior and try find the input that maximizes the accuracy. Hence, it performs significantly better than line-search or random search.

For a more detailed introduction, you can read: http://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.

Now, you could use Spearmint executing your R code through a very simple python wrapper which takes as input the parameters and just returns the accuracy https://github.com/HIPS/Spearmint. Another great package that I like is PyBO https://github.com/mwhoffman/pybo/.

| cite | improve this answer | |
  • $\begingroup$ I have a question about introduction of feature transformations. I think if you use tree-based models, like xgboost, you don't have to do and variable scaling. Also, any functional transformation on the original set of features is also unnecessary, since tree-based models is able to capture almost any complex non-linear patterns. For example, if you squared a feature, this can be captured by a set of step functions. So I am wondering why for tree-based models we need to do any feature transformation. I think tune the hyper parameters in xgboost is effectively doing feature transformation $\endgroup$ – KevinKim Mar 24 '17 at 4:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.